Oct 30 22:04:03.020561 ip-10-0-130-77 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Oct 30 22:04:03.020572 ip-10-0-130-77 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Oct 30 22:04:03.020579 ip-10-0-130-77 systemd[1]: kubelet.service: Failed with result 'resources'.
Oct 30 22:04:03.020817 ip-10-0-130-77 systemd[1]: Failed to start Kubernetes Kubelet.
Oct 30 22:04:13.063051 ip-10-0-130-77 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Oct 30 22:04:13.063066 ip-10-0-130-77 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 2d4fffbff7064c4dbe0adc4764b67222 --
Oct 30 22:06:16.133325 ip-10-0-130-77 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 22:06:16.539415 ip-10-0-130-77 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 22:06:16.539415 ip-10-0-130-77 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 22:06:16.539415 ip-10-0-130-77 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 22:06:16.539415 ip-10-0-130-77 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 22:06:16.542731 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.542653    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 22:06:16.542731 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.542731    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 22:06:16.542801 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.542736    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 22:06:16.542801 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.542739    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 22:06:16.542801 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.542742    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 22:06:16.542801 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.542745    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 22:06:16.542801 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.542748    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 22:06:16.542801 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.542751    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 22:06:16.542801 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.542754    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 22:06:16.542801 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.542757    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 22:06:16.542801 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.542760    2579 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 22:06:16.542801 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.542763    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 22:06:16.542801 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.542767    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 22:06:16.542801 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.542769    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 22:06:16.542801 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.542772    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 22:06:16.542801 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.542774    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 22:06:16.542801 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.542777    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 22:06:16.542801 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.542780    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 22:06:16.542801 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.542783    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 22:06:16.542801 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.542785    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 22:06:16.542801 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.542788    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 22:06:16.543261 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.542790    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 22:06:16.543261 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.542793    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 22:06:16.543261 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.542796    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 22:06:16.543261 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.542798    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 22:06:16.543261 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.542801    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 22:06:16.543261 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.542805    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 22:06:16.543261 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.542809    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 22:06:16.543261 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.542813    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 22:06:16.543261 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.542817    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 22:06:16.543261 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.542820    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 22:06:16.543261 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.542823    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 22:06:16.543261 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.542825    2579 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 22:06:16.543261 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.542828    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 22:06:16.543261 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.542831    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 22:06:16.543261 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.542833    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 22:06:16.543261 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.542836    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 22:06:16.543261 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.542838    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 22:06:16.543261 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.542841    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 22:06:16.543261 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.542844    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 22:06:16.543261 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.542847    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 22:06:16.543731 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.542849    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 22:06:16.543731 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.542852    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 22:06:16.543731 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.542854    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 22:06:16.543731 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.542857    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 22:06:16.543731 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.542859    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 22:06:16.543731 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.542863    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 22:06:16.543731 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.542867    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 22:06:16.543731 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.542870    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 22:06:16.543731 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.542873    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 22:06:16.543731 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.542875    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 22:06:16.543731 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.542878    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 22:06:16.543731 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.542882    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 22:06:16.543731 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.542884    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 22:06:16.543731 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.542887    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 22:06:16.543731 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.542891    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 22:06:16.543731 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.542894    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 22:06:16.543731 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.542896    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 22:06:16.543731 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.542899    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 22:06:16.543731 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.542901    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 22:06:16.544191 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.542904    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 22:06:16.544191 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.542907    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 22:06:16.544191 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.542909    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 22:06:16.544191 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.542912    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 22:06:16.544191 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.542914    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 22:06:16.544191 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.542917    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 22:06:16.544191 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.542919    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 22:06:16.544191 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.542922    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 22:06:16.544191 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.542924    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 22:06:16.544191 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.542927    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 22:06:16.544191 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.542930    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 22:06:16.544191 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.542933    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 22:06:16.544191 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.542936    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 22:06:16.544191 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.542938    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 22:06:16.544191 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.542941    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 22:06:16.544191 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.542944    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 22:06:16.544191 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.542946    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 22:06:16.544191 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.542949    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 22:06:16.544191 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.542951    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 22:06:16.544191 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.542954    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 22:06:16.544191 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.542956    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 22:06:16.544698 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.542959    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 22:06:16.544698 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.542962    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 22:06:16.544698 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.542964    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 22:06:16.544698 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.542967    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 22:06:16.544698 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.542970    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 22:06:16.544698 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.542973    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 22:06:16.544698 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.542975    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 22:06:16.544698 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.542978    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 22:06:16.544698 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.542980    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 22:06:16.544698 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.542983    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 22:06:16.544698 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.542985    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 22:06:16.544698 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.542988    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 22:06:16.544698 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.542990    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 22:06:16.544698 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.542994    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 22:06:16.544698 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.542997    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 22:06:16.544698 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.543000    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 22:06:16.544698 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.543002    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 22:06:16.544698 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.543005    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 22:06:16.544698 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.543007    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 22:06:16.545161 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.543010    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 22:06:16.545161 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.543013    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 22:06:16.545161 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.543015    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 22:06:16.545161 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.543018    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 22:06:16.545161 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.543022    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 22:06:16.545161 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.543025    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 22:06:16.545161 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.543834    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 22:06:16.545161 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.543842    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 22:06:16.545161 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.543846    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 22:06:16.545161 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.543850    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 22:06:16.545161 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.543853    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 22:06:16.545161 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.543855    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 22:06:16.545161 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.543859    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 22:06:16.545161 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.543861    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 22:06:16.545161 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.543864    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 22:06:16.545161 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.543868    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 22:06:16.545161 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.543871    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 22:06:16.545161 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.543873    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 22:06:16.545161 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.543877    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 22:06:16.545662 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.543893    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 22:06:16.545662 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.543897    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 22:06:16.545662 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.543900    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 22:06:16.545662 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.543904    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 22:06:16.545662 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.543908    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 22:06:16.545662 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.543911    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 22:06:16.545662 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.543914    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 22:06:16.545662 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.543916    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 22:06:16.545662 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.543923    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 22:06:16.545662 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.543926    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 22:06:16.545662 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.543929    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 22:06:16.545662 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.543932    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 22:06:16.545662 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.543934    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 22:06:16.545662 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.543937    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 22:06:16.545662 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.543940    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 22:06:16.545662 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.543942    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 22:06:16.545662 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.543945    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 22:06:16.545662 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.543947    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 22:06:16.545662 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.543950    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 22:06:16.545662 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.543953    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 22:06:16.546160 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.543955    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 22:06:16.546160 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.543958    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 22:06:16.546160 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.543960    2579 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 22:06:16.546160 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.543963    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 22:06:16.546160 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.543965    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 22:06:16.546160 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.543968    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 22:06:16.546160 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.543970    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 22:06:16.546160 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.543973    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 22:06:16.546160 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.543976    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 22:06:16.546160 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.543979    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 22:06:16.546160 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.543981    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 22:06:16.546160 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.543986    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 22:06:16.546160 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.543988    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 22:06:16.546160 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.543991    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 22:06:16.546160 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.543993    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 22:06:16.546160 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.543996    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 22:06:16.546160 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.543999    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 22:06:16.546160 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.544001    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 22:06:16.546160 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.544005    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 22:06:16.546612 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.544009    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 22:06:16.546612 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.544012    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 22:06:16.546612 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.544015    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 22:06:16.546612 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.544018    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 22:06:16.546612 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.544020    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 22:06:16.546612 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.544023    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 22:06:16.546612 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.544025    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 22:06:16.546612 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.544028    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 22:06:16.546612 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.544030    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 22:06:16.546612 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.544033    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 22:06:16.546612 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.544036    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 22:06:16.546612 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.544052    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 22:06:16.546612 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.544055    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 22:06:16.546612 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.544057    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 22:06:16.546612 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.544060    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 22:06:16.546612 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.544062    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 22:06:16.546612 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.544065    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 22:06:16.546612 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.544068    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 22:06:16.546612 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.544072    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 22:06:16.547064 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.544076    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 22:06:16.547064 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.544080    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 22:06:16.547064 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.544083    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 22:06:16.547064 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.544086    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 22:06:16.547064 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.544088    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 22:06:16.547064 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.544091    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 22:06:16.547064 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.544094    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 22:06:16.547064 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.544097    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 22:06:16.547064 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.544100    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 22:06:16.547064 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.544103    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 22:06:16.547064 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.544105    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 22:06:16.547064 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.544108    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 22:06:16.547064 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.544111    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 22:06:16.547064 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.544113    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 22:06:16.547064 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.544116    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 22:06:16.547064 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.544118    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 22:06:16.547064 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.544121    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 22:06:16.547064 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.544124    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 22:06:16.547064 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.544127    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 22:06:16.547064 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.544130    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 22:06:16.547543 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.544133    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 22:06:16.547543 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.544135    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 22:06:16.547543 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.544137    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 22:06:16.547543 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.544140    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 22:06:16.547543 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.544142    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 22:06:16.547543 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.544145    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 22:06:16.547543 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.544147    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 22:06:16.547543 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.544150    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 22:06:16.547543 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.544152    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 22:06:16.547543 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.544155    2579 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 22:06:16.547543 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.544158    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 22:06:16.547543 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.544160    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 22:06:16.547543 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.544163    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 22:06:16.547543 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.544166    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 22:06:16.547543 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.544169    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 22:06:16.547543 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.545680    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 22:06:16.547543 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.545687    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 22:06:16.547543 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.545691    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 22:06:16.547543 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.545694    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 22:06:16.548215 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.545697    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 22:06:16.548215 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.545700    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 22:06:16.548215 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.545702    2579 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 22:06:16.548215 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.545705    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 22:06:16.548215 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.545708    2579 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 22:06:16.548215 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.545710    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 22:06:16.548215 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.545713    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 22:06:16.548215 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.545716    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 22:06:16.548215 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.545719    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 22:06:16.548215 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.545722    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 22:06:16.548215 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.545724    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 22:06:16.548215 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.545727    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 22:06:16.548215 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.545729    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 22:06:16.548215 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.545732    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 22:06:16.548215 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.545735    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 22:06:16.548215 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.545737    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 22:06:16.548215 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.545740    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 22:06:16.548215 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.545742    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 22:06:16.548215 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.545745    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 22:06:16.548846 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.545747    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 22:06:16.548846 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.545749    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 22:06:16.548846 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.545752    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 22:06:16.548846 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.545755    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 22:06:16.548846 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.545757    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 22:06:16.548846 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.545760    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 22:06:16.548846 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.545763    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 22:06:16.548846 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.545765    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 22:06:16.548846 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.545768    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 22:06:16.548846 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.545770    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 22:06:16.548846 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.545773    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 22:06:16.548846 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.545776    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 22:06:16.548846 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.545779    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 22:06:16.548846 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.545782    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 22:06:16.548846 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.545784    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 22:06:16.548846 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.545787    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 22:06:16.548846 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.545789    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 22:06:16.548846 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.545792    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 22:06:16.548846 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.545794    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 22:06:16.548846 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.545797    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 22:06:16.549511 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.545799    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 22:06:16.549511 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.545802    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 22:06:16.549511 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.545804    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 22:06:16.549511 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.545808    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 22:06:16.549511 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.545810    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 22:06:16.549511 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.545814    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 22:06:16.549511 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.545816    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 22:06:16.549511 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.545818    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 22:06:16.549511 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.545822    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 22:06:16.549511 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.545824    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 22:06:16.549511 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.545827    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 22:06:16.549511 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.545829    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 22:06:16.549511 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.545832    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 22:06:16.549511 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.545834    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 22:06:16.549511 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.545837    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 22:06:16.549511 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.545839    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 22:06:16.549511 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.545842    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 22:06:16.549511 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.545845    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 22:06:16.549511 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.545847    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 22:06:16.549511 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.545850    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 22:06:16.549984 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.545854    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 22:06:16.549984 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.545859    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 22:06:16.549984 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.545862    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 22:06:16.549984 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.545864    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 22:06:16.549984 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.545867    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 22:06:16.549984 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.545870    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 22:06:16.549984 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.545872    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 22:06:16.549984 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.545875    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 22:06:16.549984 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.545878    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 22:06:16.549984 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.545882    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 22:06:16.549984 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.545885    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 22:06:16.549984 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.545888    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 22:06:16.549984 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.545891    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 22:06:16.549984 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.545893    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 22:06:16.549984 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.545896    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 22:06:16.549984 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.545898    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 22:06:16.549984 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.545901    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 22:06:16.549984 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.545904    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 22:06:16.549984 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.545906    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 22:06:16.550489 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.545909    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 22:06:16.550489 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.545912    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 22:06:16.550489 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.545914    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 22:06:16.550489 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.545918    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 22:06:16.550489 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.545920    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 22:06:16.550489 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.545923    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 22:06:16.550489 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.545925    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 22:06:16.550489 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.545928    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 22:06:16.550489 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.545930    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 22:06:16.550489 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.545933    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 22:06:16.550489 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.545939    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 22:06:16.550489 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.545942    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 22:06:16.550489 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.545944    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 22:06:16.550489 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.545946    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 22:06:16.550489 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.545949    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 22:06:16.550489 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.545952    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 22:06:16.550489 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.545954    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 22:06:16.550489 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.545957    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 22:06:16.550489 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.545959    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 22:06:16.550489 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.545962    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 22:06:16.550959 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.545964    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 22:06:16.550959 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.545967    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 22:06:16.550959 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.545969    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 22:06:16.550959 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.545972    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 22:06:16.550959 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.546978    2579 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 22:06:16.550959 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.546993    2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 22:06:16.550959 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.546999    2579 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 22:06:16.550959 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547004    2579 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 22:06:16.550959 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547009    2579 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 22:06:16.550959 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547012    2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 22:06:16.550959 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547018    2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 22:06:16.550959 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547022    2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 22:06:16.550959 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547026    2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 22:06:16.550959 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547029    2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 22:06:16.550959 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547033    2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 22:06:16.550959 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547036    2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 22:06:16.550959 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547049    2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 22:06:16.550959 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547052    2579 flags.go:64] FLAG: --cgroup-root=""
Oct 30 22:06:16.550959 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547055    2579 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 22:06:16.550959 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547058    2579 flags.go:64] FLAG: --client-ca-file=""
Oct 30 22:06:16.550959 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547061    2579 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 22:06:16.550959 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547065    2579 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 22:06:16.550959 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547073    2579 flags.go:64] FLAG: --cluster-domain=""
Oct 30 22:06:16.551506 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547076    2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 22:06:16.551506 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547080    2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 22:06:16.551506 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547083    2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 22:06:16.551506 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547086    2579 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 22:06:16.551506 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547090    2579 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 22:06:16.551506 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547093    2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 22:06:16.551506 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547097    2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 22:06:16.551506 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547100    2579 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 22:06:16.551506 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547103    2579 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 22:06:16.551506 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547106    2579 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 22:06:16.551506 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547109    2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 22:06:16.551506 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547112    2579 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 22:06:16.551506 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547115    2579 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 22:06:16.551506 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547120    2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 22:06:16.551506 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547123    2579 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 22:06:16.551506 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547126    2579 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 22:06:16.551506 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547128    2579 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 22:06:16.551506 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547131    2579 flags.go:64] FLAG: --enable-server="true"
Oct 30 22:06:16.551506 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547134    2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 22:06:16.551506 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547139    2579 flags.go:64] FLAG: --event-burst="100"
Oct 30 22:06:16.551506 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547142    2579 flags.go:64] FLAG: --event-qps="50"
Oct 30 22:06:16.551506 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547145    2579 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 22:06:16.551506 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547148    2579 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 22:06:16.551506 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547151    2579 flags.go:64] FLAG: --eviction-hard=""
Oct 30 22:06:16.552128 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547156    2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 22:06:16.552128 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547158    2579 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 22:06:16.552128 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547162    2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 22:06:16.552128 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547165    2579 flags.go:64] FLAG: --eviction-soft=""
Oct 30 22:06:16.552128 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547168    2579 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 22:06:16.552128 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547171    2579 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 22:06:16.552128 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547173    2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 22:06:16.552128 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547176    2579 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 22:06:16.552128 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547180    2579 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 22:06:16.552128 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547183    2579 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 22:06:16.552128 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547185    2579 flags.go:64] FLAG: --feature-gates=""
Oct 30 22:06:16.552128 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547189    2579 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 22:06:16.552128 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547192    2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 22:06:16.552128 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547196    2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 22:06:16.552128 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547199    2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 22:06:16.552128 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547202    2579 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 22:06:16.552128 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547205    2579 flags.go:64] FLAG: --help="false"
Oct 30 22:06:16.552128 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547208    2579 flags.go:64] FLAG: --hostname-override="ip-10-0-130-77.ec2.internal"
Oct 30 22:06:16.552128 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547211    2579 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 22:06:16.552128 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547214    2579 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 22:06:16.552128 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547217    2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 22:06:16.552128 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547220    2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 22:06:16.552128 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547223    2579 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 22:06:16.552800 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547226    2579 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 22:06:16.552800 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547229    2579 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 22:06:16.552800 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547232    2579 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 22:06:16.552800 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547235    2579 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 22:06:16.552800 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547238    2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 22:06:16.552800 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547242    2579 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 22:06:16.552800 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547245    2579 flags.go:64] FLAG: --kube-reserved=""
Oct 30 22:06:16.552800 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547250    2579 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 22:06:16.552800 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547253    2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 22:06:16.552800 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547256    2579 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 22:06:16.552800 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547259    2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 22:06:16.552800 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547262    2579 flags.go:64] FLAG: --lock-file=""
Oct 30 22:06:16.552800 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547265    2579 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 22:06:16.552800 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547268    2579 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 22:06:16.552800 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547271    2579 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 22:06:16.552800 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547276    2579 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 22:06:16.552800 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547279    2579 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 22:06:16.552800 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547282    2579 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 22:06:16.552800 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547285    2579 flags.go:64] FLAG: --logging-format="text"
Oct 30 22:06:16.552800 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547288    2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 22:06:16.552800 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547291    2579 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 22:06:16.552800 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547295    2579 flags.go:64] FLAG: --manifest-url=""
Oct 30 22:06:16.552800 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547298    2579 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 22:06:16.552800 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547303    2579 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 22:06:16.552800 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547306    2579 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 22:06:16.553438 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547310    2579 flags.go:64] FLAG: --max-pods="110"
Oct 30 22:06:16.553438 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547313    2579 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 22:06:16.553438 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547316    2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 22:06:16.553438 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547318    2579 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 22:06:16.553438 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547321    2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 22:06:16.553438 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547325    2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 22:06:16.553438 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547328    2579 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 22:06:16.553438 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547331    2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 22:06:16.553438 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547338    2579 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 22:06:16.553438 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547341    2579 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 22:06:16.553438 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547343    2579 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 22:06:16.553438 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547347    2579 flags.go:64] FLAG: --pod-cidr=""
Oct 30 22:06:16.553438 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547349    2579 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 22:06:16.553438 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547352    2579 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 22:06:16.553438 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547357    2579 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 22:06:16.553438 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547360    2579 flags.go:64] FLAG: --port="10250"
Oct 30 22:06:16.553438 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547363    2579 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 22:06:16.553438 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547366    2579 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-00158a26bf59842fb"
Oct 30 22:06:16.553438 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547370    2579 flags.go:64] FLAG: --qos-reserved=""
Oct 30 22:06:16.553438 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547372    2579 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 22:06:16.553438 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547375    2579 flags.go:64] FLAG: --register-node="true"
Oct 30 22:06:16.553438 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547378    2579 flags.go:64] FLAG: --register-with-taints=""
Oct 30 22:06:16.553438 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547382    2579 flags.go:64] FLAG: --registry-burst="10"
Oct 30 22:06:16.553438 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547385    2579 flags.go:64] FLAG: --registry-qps="5"
Oct 30 22:06:16.553438 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547388    2579 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 22:06:16.553438 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547391    2579 flags.go:64] FLAG: --reserved-memory=""
Oct 30 22:06:16.554104 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547395    2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 22:06:16.554104 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547398    2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 22:06:16.554104 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547401    2579 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 22:06:16.554104 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547403    2579 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 22:06:16.554104 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547406    2579 flags.go:64] FLAG: --runonce="false"
Oct 30 22:06:16.554104 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547409    2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 22:06:16.554104 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547413    2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 22:06:16.554104 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547416    2579 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 22:06:16.554104 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547419    2579 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 22:06:16.554104 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547422    2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 22:06:16.554104 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547425    2579 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 22:06:16.554104 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547428    2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 22:06:16.554104 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547431    2579 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 22:06:16.554104 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547434    2579 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 22:06:16.554104 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547437    2579 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 22:06:16.554104 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547439    2579 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 22:06:16.554104 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547442    2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 22:06:16.554104 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547445    2579 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 22:06:16.554104 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547448    2579 flags.go:64] FLAG: --system-cgroups=""
Oct 30 22:06:16.554104 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547451    2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 22:06:16.554104 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547457    2579 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 22:06:16.554104 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547460    2579 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 22:06:16.554104 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547464    2579 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 22:06:16.554104 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547468    2579 flags.go:64] FLAG: --tls-min-version=""
Oct 30 22:06:16.554104 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547471    2579 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 22:06:16.554755 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547474    2579 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 22:06:16.554755 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547477    2579 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 22:06:16.554755 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547480    2579 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 22:06:16.554755 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547483    2579 flags.go:64] FLAG: --v="2"
Oct 30 22:06:16.554755 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547488    2579 flags.go:64] FLAG: --version="false"
Oct 30 22:06:16.554755 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547493    2579 flags.go:64] FLAG: --vmodule=""
Oct 30 22:06:16.554755 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547497    2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 22:06:16.554755 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.547500    2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 22:06:16.554755 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.548701    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 22:06:16.554755 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.548721    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 22:06:16.554755 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.548728    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 22:06:16.554755 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.548735    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 22:06:16.554755 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.548740    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 22:06:16.554755 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.548745    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 22:06:16.554755 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.548749    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 22:06:16.554755 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.548754    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 22:06:16.554755 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.548766    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 22:06:16.554755 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.548771    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 22:06:16.554755 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.548776    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 22:06:16.554755 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.548780    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 22:06:16.554755 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.548785    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 22:06:16.555291 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.548789    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 22:06:16.555291 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.548794    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 22:06:16.555291 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.548798    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 22:06:16.555291 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.548805    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 22:06:16.555291 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.548813    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 22:06:16.555291 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.548818    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 22:06:16.555291 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.548823    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 22:06:16.555291 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.548827    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 22:06:16.555291 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.548837    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 22:06:16.555291 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.548846    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 22:06:16.555291 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549070    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 22:06:16.555291 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549081    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 22:06:16.555291 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549086    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 22:06:16.555291 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549089    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 22:06:16.555291 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549093    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 22:06:16.555291 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549097    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 22:06:16.555291 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549100    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 22:06:16.555291 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549103    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 22:06:16.555291 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549106    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 22:06:16.555749 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549108    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 22:06:16.555749 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549111    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 22:06:16.555749 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549119    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 22:06:16.555749 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549122    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 22:06:16.555749 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549125    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 22:06:16.555749 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549127    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 22:06:16.555749 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549131    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 22:06:16.555749 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549134    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 22:06:16.555749 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549137    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 22:06:16.555749 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549139    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 22:06:16.555749 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549142    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 22:06:16.555749 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549145    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 22:06:16.555749 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549147    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 22:06:16.555749 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549150    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 22:06:16.555749 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549153    2579 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 22:06:16.555749 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549156    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 22:06:16.555749 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549159    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 22:06:16.555749 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549161    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 22:06:16.555749 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549164    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 22:06:16.555749 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549167    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 22:06:16.556236 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549170    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 22:06:16.556236 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549173    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 22:06:16.556236 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549176    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 22:06:16.556236 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549179    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 22:06:16.556236 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549182    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 22:06:16.556236 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549185    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 22:06:16.556236 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549187    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 22:06:16.556236 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549190    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 22:06:16.556236 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549193    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 22:06:16.556236 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549195    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 22:06:16.556236 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549198    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 22:06:16.556236 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549201    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 22:06:16.556236 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549206    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 22:06:16.556236 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549208    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 22:06:16.556236 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549211    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 22:06:16.556236 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549214    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 22:06:16.556236 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549217    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 22:06:16.556236 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549220    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 22:06:16.556236 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549223    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 22:06:16.556695 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549225    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 22:06:16.556695 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549228    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 22:06:16.556695 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549231    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 22:06:16.556695 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549234    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 22:06:16.556695 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549241    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 22:06:16.556695 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549244    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 22:06:16.556695 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549248    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 22:06:16.556695 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549251    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 22:06:16.556695 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549254    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 22:06:16.556695 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549257    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 22:06:16.556695 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549259    2579 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 22:06:16.556695 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549262    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 22:06:16.556695 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549264    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 22:06:16.556695 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549268    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 22:06:16.556695 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549270    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 22:06:16.556695 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549273    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 22:06:16.556695 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549276    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 22:06:16.556695 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549278    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 22:06:16.556695 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549281    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 22:06:16.557165 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549284    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 22:06:16.557165 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549286    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 22:06:16.557165 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549289    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 22:06:16.557165 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549291    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 22:06:16.557165 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549294    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 22:06:16.557165 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549296    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 22:06:16.557165 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549299    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 22:06:16.557165 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549302    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 22:06:16.557165 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549305    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 22:06:16.557165 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549307    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 22:06:16.557165 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549310    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 22:06:16.557165 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549312    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 22:06:16.557165 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549315    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 22:06:16.557165 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549318    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 22:06:16.557165 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549320    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 22:06:16.557165 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549323    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 22:06:16.557165 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.549358    2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 22:06:16.557567 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549397    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 22:06:16.557567 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549402    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 22:06:16.557567 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549405    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 22:06:16.557567 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549408    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 22:06:16.557567 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549411    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 22:06:16.557567 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549414    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 22:06:16.557567 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549417    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 22:06:16.557567 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549420    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 22:06:16.557567 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549424    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 22:06:16.557567 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549429    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 22:06:16.557567 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549432    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 22:06:16.557567 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549435    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 22:06:16.557567 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549438    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 22:06:16.557567 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549441    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 22:06:16.557567 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549443    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 22:06:16.557567 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549446    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 22:06:16.557567 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549449    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 22:06:16.557567 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549451    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 22:06:16.557567 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549454    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 22:06:16.558018 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549456    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 22:06:16.558018 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549459    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 22:06:16.558018 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549462    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 22:06:16.558018 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549464    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 22:06:16.558018 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549467    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 22:06:16.558018 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549470    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 22:06:16.558018 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549472    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 22:06:16.558018 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549475    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 22:06:16.558018 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549477    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 22:06:16.558018 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549480    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 22:06:16.558018 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549482    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 22:06:16.558018 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549485    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 22:06:16.558018 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549487    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 22:06:16.558018 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549490    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 22:06:16.558018 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549494    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 22:06:16.558018 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549496    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 22:06:16.558018 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549499    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 22:06:16.558018 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549502    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 22:06:16.558018 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549505    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 22:06:16.558018 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549508    2579 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 22:06:16.558502 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549510    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 22:06:16.558502 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549513    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 22:06:16.558502 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549516    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 22:06:16.558502 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549519    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 22:06:16.558502 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549522    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 22:06:16.558502 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549525    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 22:06:16.558502 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549527    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 22:06:16.558502 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549530    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 22:06:16.558502 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549532    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 22:06:16.558502 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549535    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 22:06:16.558502 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549538    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 22:06:16.558502 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549540    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 22:06:16.558502 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549543    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 22:06:16.558502 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549545    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 22:06:16.558502 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549548    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 22:06:16.558502 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549550    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 22:06:16.558502 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549553    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 22:06:16.558502 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549556    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 22:06:16.558502 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549558    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 22:06:16.558956 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549561    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 22:06:16.558956 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549564    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 22:06:16.558956 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549567    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 22:06:16.558956 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549570    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 22:06:16.558956 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549572    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 22:06:16.558956 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549575    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 22:06:16.558956 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549577    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 22:06:16.558956 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549581    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 22:06:16.558956 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549584    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 22:06:16.558956 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549587    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 22:06:16.558956 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549591    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 22:06:16.558956 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549593    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 22:06:16.558956 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549596    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 22:06:16.558956 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549599    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 22:06:16.558956 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549601    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 22:06:16.558956 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549604    2579 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 22:06:16.558956 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549606    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 22:06:16.558956 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549609    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 22:06:16.558956 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549612    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 22:06:16.559429 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549615    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 22:06:16.559429 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549617    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 22:06:16.559429 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549620    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 22:06:16.559429 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549623    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 22:06:16.559429 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549625    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 22:06:16.559429 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549628    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 22:06:16.559429 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549630    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 22:06:16.559429 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549633    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 22:06:16.559429 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549635    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 22:06:16.559429 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549638    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 22:06:16.559429 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549640    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 22:06:16.559429 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549643    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 22:06:16.559429 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549647    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 22:06:16.559429 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549650    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 22:06:16.559429 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549652    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 22:06:16.559429 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549654    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 22:06:16.559429 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549657    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 22:06:16.559429 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549660    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 22:06:16.559429 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549662    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 22:06:16.559429 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549665    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 22:06:16.559940 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549668    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 22:06:16.559940 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549671    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 22:06:16.559940 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549674    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 22:06:16.559940 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549676    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 22:06:16.559940 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549679    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 22:06:16.559940 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549682    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 22:06:16.559940 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549684    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 22:06:16.559940 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549686    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 22:06:16.559940 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.549689    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 22:06:16.559940 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.556121    2579 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 22:06:16.559940 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.556134    2579 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 22:06:16.559940 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.556150    2579 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 22:06:16.559940 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.556155    2579 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 22:06:16.559940 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556209    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 22:06:16.559940 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556213    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 22:06:16.559940 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556217    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 22:06:16.559940 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556221    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 22:06:16.559940 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556224    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 22:06:16.559940 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556227    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 22:06:16.561533 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556229    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 22:06:16.561533 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556232    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 22:06:16.561533 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556234    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 22:06:16.561533 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556237    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 22:06:16.561533 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556240    2579 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 22:06:16.561533 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556243    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 22:06:16.561533 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556246    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 22:06:16.561533 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556249    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 22:06:16.561533 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556252    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 22:06:16.561533 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556255    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 22:06:16.561533 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556257    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 22:06:16.561533 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556260    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 22:06:16.561533 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556263    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 22:06:16.561533 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556266    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 22:06:16.561533 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556268    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 22:06:16.561533 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556271    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 22:06:16.561533 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556273    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 22:06:16.561533 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556276    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 22:06:16.561533 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556278    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 22:06:16.561533 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556281    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 22:06:16.562009 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556283    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 22:06:16.562009 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556286    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 22:06:16.562009 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556288    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 22:06:16.562009 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556293    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 22:06:16.562009 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556296    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 22:06:16.562009 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556298    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 22:06:16.562009 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556301    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 22:06:16.562009 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556303    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 22:06:16.562009 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556306    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 22:06:16.562009 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556308    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 22:06:16.562009 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556311    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 22:06:16.562009 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556313    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 22:06:16.562009 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556316    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 22:06:16.562009 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556319    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 22:06:16.562009 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556321    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 22:06:16.562009 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556324    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 22:06:16.562009 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556326    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 22:06:16.562009 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556329    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 22:06:16.562009 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556331    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 22:06:16.562466 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556334    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 22:06:16.562466 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556336    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 22:06:16.562466 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556339    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 22:06:16.562466 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556342    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 22:06:16.562466 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556344    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 22:06:16.562466 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556347    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 22:06:16.562466 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556349    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 22:06:16.562466 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556352    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 22:06:16.562466 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556354    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 22:06:16.562466 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556357    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 22:06:16.562466 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556359    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 22:06:16.562466 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556362    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 22:06:16.562466 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556364    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 22:06:16.562466 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556367    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 22:06:16.562466 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556369    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 22:06:16.562466 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556373    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 22:06:16.562466 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556377    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 22:06:16.562466 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556380    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 22:06:16.562466 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556383    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 22:06:16.562922 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556386    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 22:06:16.562922 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556388    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 22:06:16.562922 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556391    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 22:06:16.562922 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556393    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 22:06:16.562922 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556396    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 22:06:16.562922 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556398    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 22:06:16.562922 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556401    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 22:06:16.562922 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556403    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 22:06:16.562922 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556406    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 22:06:16.562922 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556408    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 22:06:16.562922 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556411    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 22:06:16.562922 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556413    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 22:06:16.562922 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556416    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 22:06:16.562922 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556418    2579 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 22:06:16.562922 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556420    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 22:06:16.562922 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556423    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 22:06:16.562922 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556425    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 22:06:16.562922 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556428    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 22:06:16.562922 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556430    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 22:06:16.562922 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556433    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 22:06:16.563415 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556435    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 22:06:16.563415 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556438    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 22:06:16.563415 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556440    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 22:06:16.563415 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556443    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 22:06:16.563415 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556445    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 22:06:16.563415 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556448    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 22:06:16.563415 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556450    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 22:06:16.563415 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556453    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 22:06:16.563415 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556455    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 22:06:16.563415 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556458    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 22:06:16.563415 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556461    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 22:06:16.563415 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556464    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 22:06:16.563415 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556467    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 22:06:16.563415 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556469    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 22:06:16.563415 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556473    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 22:06:16.563415 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556477    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 22:06:16.563415 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556480    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 22:06:16.563415 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556483    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 22:06:16.563415 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556486    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 22:06:16.563864 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556488    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 22:06:16.563864 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556491    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 22:06:16.563864 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556493    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 22:06:16.563864 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.556521    2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 22:06:16.563864 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556665    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 22:06:16.563864 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556670    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 22:06:16.563864 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556673    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 22:06:16.563864 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556676    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 22:06:16.563864 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556678    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 22:06:16.563864 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556681    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 22:06:16.563864 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556683    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 22:06:16.563864 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556686    2579 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 22:06:16.563864 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556688    2579 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 22:06:16.563864 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556691    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 22:06:16.563864 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556693    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 22:06:16.563864 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556696    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 22:06:16.564255 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556699    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 22:06:16.564255 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556701    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 22:06:16.564255 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556704    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 22:06:16.564255 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556706    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 22:06:16.564255 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556709    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 22:06:16.564255 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556712    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 22:06:16.564255 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556715    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 22:06:16.564255 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556717    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 22:06:16.564255 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556720    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 22:06:16.564255 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556723    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 22:06:16.564255 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556726    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 22:06:16.564255 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556728    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 22:06:16.564255 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556731    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 22:06:16.564255 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556733    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 22:06:16.564255 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556736    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 22:06:16.564255 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556738    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 22:06:16.564255 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556742    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 22:06:16.564255 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556746    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 22:06:16.564255 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556749    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 22:06:16.564717 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556752    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 22:06:16.564717 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556756    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 22:06:16.564717 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556759    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 22:06:16.564717 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556763    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 22:06:16.564717 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556766    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 22:06:16.564717 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556768    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 22:06:16.564717 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556771    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 22:06:16.564717 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556773    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 22:06:16.564717 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556776    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 22:06:16.564717 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556779    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 22:06:16.564717 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556781    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 22:06:16.564717 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556783    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 22:06:16.564717 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556786    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 22:06:16.564717 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556788    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 22:06:16.564717 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556791    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 22:06:16.564717 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556794    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 22:06:16.564717 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556796    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 22:06:16.564717 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556799    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 22:06:16.564717 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556801    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 22:06:16.565185 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556804    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 22:06:16.565185 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556806    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 22:06:16.565185 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556809    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 22:06:16.565185 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556811    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 22:06:16.565185 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556814    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 22:06:16.565185 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556817    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 22:06:16.565185 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556820    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 22:06:16.565185 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556822    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 22:06:16.565185 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556824    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 22:06:16.565185 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556827    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 22:06:16.565185 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556830    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 22:06:16.565185 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556832    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 22:06:16.565185 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556835    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 22:06:16.565185 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556837    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 22:06:16.565185 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556840    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 22:06:16.565185 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556842    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 22:06:16.565185 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556844    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 22:06:16.565185 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556852    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 22:06:16.565185 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556855    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 22:06:16.565694 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556857    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 22:06:16.565694 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556860    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 22:06:16.565694 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556862    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 22:06:16.565694 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556865    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 22:06:16.565694 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556867    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 22:06:16.565694 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556870    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 22:06:16.565694 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556872    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 22:06:16.565694 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556875    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 22:06:16.565694 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556878    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 22:06:16.565694 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556880    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 22:06:16.565694 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556883    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 22:06:16.565694 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556887    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 22:06:16.565694 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556890    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 22:06:16.565694 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556892    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 22:06:16.565694 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556895    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 22:06:16.565694 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556898    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 22:06:16.565694 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556900    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 22:06:16.565694 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556903    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 22:06:16.565694 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556906    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 22:06:16.565694 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556909    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 22:06:16.566179 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556912    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 22:06:16.566179 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556914    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 22:06:16.566179 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556917    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 22:06:16.566179 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556919    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 22:06:16.566179 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556922    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 22:06:16.566179 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556924    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 22:06:16.566179 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556927    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 22:06:16.566179 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556929    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 22:06:16.566179 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556932    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 22:06:16.566179 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556934    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 22:06:16.566179 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556938    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 22:06:16.566179 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556941    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 22:06:16.566179 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556944    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 22:06:16.566179 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556946    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 22:06:16.566179 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556948    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 22:06:16.566179 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556951    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 22:06:16.566179 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.556954    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 22:06:16.566579 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.556976    2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 22:06:16.566579 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.557008    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 22:06:16.566579 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.557012    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 22:06:16.566579 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.557014    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 22:06:16.566579 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.557017    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 22:06:16.566579 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.557020    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 22:06:16.566579 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.557022    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 22:06:16.566579 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.557026    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 22:06:16.566579 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.557029    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 22:06:16.566579 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.557031    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 22:06:16.566579 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.557034    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 22:06:16.566579 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.557036    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 22:06:16.566579 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.557052    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 22:06:16.566579 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.557055    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 22:06:16.566579 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.557058    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 22:06:16.566579 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.557061    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 22:06:16.567014 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.557063    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 22:06:16.567014 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.557066    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 22:06:16.567014 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.557068    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 22:06:16.567014 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.557071    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 22:06:16.567014 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.557074    2579 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 22:06:16.567014 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.557076    2579 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 22:06:16.567014 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.557079    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 22:06:16.567014 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.557081    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 22:06:16.567014 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.557084    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 22:06:16.567014 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.557088    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 22:06:16.567014 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.557091    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 22:06:16.567014 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.557094    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 22:06:16.567014 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.557096    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 22:06:16.567014 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.557099    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 22:06:16.567014 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.557102    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 22:06:16.567014 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.557104    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 22:06:16.567014 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.557107    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 22:06:16.567014 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.557110    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 22:06:16.567014 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.557112    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 22:06:16.567014 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.557115    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 22:06:16.567698 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.557118    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 22:06:16.567698 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.557120    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 22:06:16.567698 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.557123    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 22:06:16.567698 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.557127    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 22:06:16.567698 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.557130    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 22:06:16.567698 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.557133    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 22:06:16.567698 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.557137    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 22:06:16.567698 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.557140    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 22:06:16.567698 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.557144    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 22:06:16.567698 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.557148    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 22:06:16.567698 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.557151    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 22:06:16.567698 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.557154    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 22:06:16.567698 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.557157    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 22:06:16.567698 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.557160    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 22:06:16.567698 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.557162    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 22:06:16.567698 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.557165    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 22:06:16.567698 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.557168    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 22:06:16.567698 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.557171    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 22:06:16.567698 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.557174    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 22:06:16.568318 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.557176    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 22:06:16.568318 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.557179    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 22:06:16.568318 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.557183    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 22:06:16.568318 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.557185    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 22:06:16.568318 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.557188    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 22:06:16.568318 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.557191    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 22:06:16.568318 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.557193    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 22:06:16.568318 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.557196    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 22:06:16.568318 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.557198    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 22:06:16.568318 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.557201    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 22:06:16.568318 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.557203    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 22:06:16.568318 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.557206    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 22:06:16.568318 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.557208    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 22:06:16.568318 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.557211    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 22:06:16.568318 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.557213    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 22:06:16.568318 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.557216    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 22:06:16.568318 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.557220    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 22:06:16.568318 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.557223    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 22:06:16.568318 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.557226    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 22:06:16.568779 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.557228    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 22:06:16.568779 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.557231    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 22:06:16.568779 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.557234    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 22:06:16.568779 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.557236    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 22:06:16.568779 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.557239    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 22:06:16.568779 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.557241    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 22:06:16.568779 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.557245    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 22:06:16.568779 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.557247    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 22:06:16.568779 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.557250    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 22:06:16.568779 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.557252    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 22:06:16.568779 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.557255    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 22:06:16.568779 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.557258    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 22:06:16.568779 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.557260    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 22:06:16.568779 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.557263    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 22:06:16.568779 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.557265    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 22:06:16.568779 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.557269    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 22:06:16.568779 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.557271    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 22:06:16.568779 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.557274    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 22:06:16.568779 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.557276    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 22:06:16.569406 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.557279    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 22:06:16.569406 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.557281    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 22:06:16.569406 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.557284    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 22:06:16.569406 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.557286    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 22:06:16.569406 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.557289    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 22:06:16.569406 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.557292    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 22:06:16.569406 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.557294    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 22:06:16.569406 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.557296    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 22:06:16.569406 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.557300    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 22:06:16.569406 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.557302    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 22:06:16.569406 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.557307    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 22:06:16.569406 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.557309    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 22:06:16.569406 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.557312    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 22:06:16.569406 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:16.557314    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 22:06:16.569406 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.557369    2579 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 22:06:16.569406 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.559389    2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 30 22:06:16.569406 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.560318    2579 server.go:1014] "Starting client certificate rotation"
Oct 30 22:06:16.569406 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.560416    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 22:06:16.569406 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.560452    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 22:06:16.570003 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.562634    2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 22:06:16.570003 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.565504    2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 22:06:16.572966 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.572948    2579 log.go:25] "Validated CRI v1 runtime API"
Oct 30 22:06:16.576424 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.576409    2579 log.go:25] "Validated CRI v1 image API"
Oct 30 22:06:16.577245 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.577222    2579 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 22:06:16.581059 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.580533    2579 fs.go:135] Filesystem UUIDs: map[6f6aa058-cd52-470d-89c7-a1636208e0c0:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 a37948a0-7809-47bd-97e0-9bbfa84028b6:/dev/nvme0n1p4]
Oct 30 22:06:16.581059 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.580556    2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 30 22:06:16.586327 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.586216    2579 manager.go:217] Machine: {Timestamp:2026-10-30 22:06:16.584658357 +0000 UTC m=+0.338012588 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100429 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2aaa8b88dcc89bc14401779433ff7e SystemUUID:ec2aaa8b-88dc-c89b-c144-01779433ff7e BootID:2d4fffbf-f706-4c4d-be0a-dc4764b67222 Filesystems:[{Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:f6:63:53:65 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:f6:63:53:65 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:72:25:0f:ca:09:9f Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 22:06:16.586968 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.586954    2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 22:06:16.587093 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.587077    2579 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 22:06:16.590312 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.590290    2579 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 22:06:16.590451 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.590313    2579 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-77.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 22:06:16.591231 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.591220    2579 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 22:06:16.591265 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.591233    2579 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 22:06:16.591265 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.591261    2579 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 22:06:16.591325 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.591275    2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 22:06:16.592144 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.592134    2579 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 22:06:16.595966 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.595954    2579 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 22:06:16.596102 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.596092    2579 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 22:06:16.596818 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.596809    2579 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 22:06:16.596853 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.596821    2579 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 22:06:16.596853 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.596835    2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 22:06:16.596853 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.596844    2579 kubelet.go:405] "Adding apiserver pod source"
Oct 30 22:06:16.596926 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.596855    2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 22:06:16.597197 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.597184    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 22:06:16.599691 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.599667    2579 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 22:06:16.605225 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.605213    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 22:06:16.606236 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.606224    2579 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 22:06:16.606273 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.606255    2579 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 22:06:16.606425 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.606410    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 22:06:16.606455 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.606440    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 22:06:16.606455 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.606453    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 22:06:16.606516 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.606464    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 22:06:16.606516 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.606477    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 22:06:16.606516 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.606490    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 22:06:16.607277 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.607268    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 22:06:16.607277 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.607278    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 22:06:16.607335 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.607285    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 22:06:16.607335 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.607292    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 22:06:16.607335 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.607300    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 22:06:16.607335 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.607311    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 22:06:16.607920 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.607910    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 22:06:16.607949 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.607923    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 22:06:16.611786 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.611771    2579 server.go:1263] "Started kubelet"
Oct 30 22:06:16.611895 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.611857    2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 22:06:16.611958 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.611911    2579 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 22:06:16.612011 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.611868    2579 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 22:06:16.612455 ip-10-0-130-77 systemd[1]: Started Kubernetes Kubelet.
Oct 30 22:06:16.613254 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.613237    2579 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 22:06:16.618324 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.618303    2579 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 22:06:16.620014 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:16.619985    2579 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 30 22:06:16.622820 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.622806    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 22:06:16.624057 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.624031    2579 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 22:06:16.624189 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:16.624166    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-77.ec2.internal\" not found"
Oct 30 22:06:16.624759 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.624745    2579 factory.go:55] Registering systemd factory
Oct 30 22:06:16.624759 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.624759    2579 factory.go:223] Registration of the systemd container factory successfully
Oct 30 22:06:16.624882 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.624861    2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 22:06:16.624930 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.624876    2579 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 22:06:16.624930 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.624896    2579 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 22:06:16.625023 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.625001    2579 factory.go:153] Registering CRI-O factory
Oct 30 22:06:16.625023 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.625014    2579 factory.go:223] Registration of the crio container factory successfully
Oct 30 22:06:16.625023 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.625017    2579 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 22:06:16.625148 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.625025    2579 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 22:06:16.625148 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.625080    2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 22:06:16.625148 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.625105    2579 factory.go:103] Registering Raw factory
Oct 30 22:06:16.625148 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.625120    2579 manager.go:1199] Started watching for new ooms in manager
Oct 30 22:06:16.626294 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.626278    2579 manager.go:319] Starting recovery of all containers
Oct 30 22:06:16.636714 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.636694    2579 manager.go:324] Recovery completed
Oct 30 22:06:16.638369 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:16.638350    2579 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": readdirent /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Oct 30 22:06:16.640367 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:16.640264    2579 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-77.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 30 22:06:16.641470 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:16.640275    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-77.ec2.internal.18e36df538d8dee9  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-77.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-77.ec2.internal,},FirstTimestamp:2026-10-30 22:06:16.611741417 +0000 UTC m=+0.365095648,LastTimestamp:2026-10-30 22:06:16.611741417 +0000 UTC m=+0.365095648,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-77.ec2.internal,}"
Oct 30 22:06:16.642873 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.642862    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 22:06:16.644714 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.644696    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-77.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 22:06:16.644791 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.644728    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-77.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 22:06:16.644791 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.644740    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-77.ec2.internal" event="NodeHasSufficientPID"
Oct 30 22:06:16.645269 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.645254    2579 cpu_manager.go:226] "Starting" policy="none"
Oct 30 22:06:16.645269 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.645266    2579 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 22:06:16.645423 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.645275    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 22:06:16.645423 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.645283    2579 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 22:06:16.646132 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.646118    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-77.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 22:06:16.646183 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.646143    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-77.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 22:06:16.646183 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.646158    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-77.ec2.internal" event="NodeHasSufficientPID"
Oct 30 22:06:16.648179 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.648158    2579 policy_none.go:50] "Start"
Oct 30 22:06:16.648266 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.648182    2579 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 22:06:16.648266 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.648196    2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 22:06:16.650343 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.650323    2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-77.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 30 22:06:16.651402 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.651390    2579 policy_none.go:44] "Start"
Oct 30 22:06:16.657821 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.657804    2579 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 22:06:16.657916 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.657858    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-rdntg"
Oct 30 22:06:16.660883 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:16.660821    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-77.ec2.internal.18e36df53acff16e  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-77.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-130-77.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-130-77.ec2.internal,},FirstTimestamp:2026-10-30 22:06:16.644710766 +0000 UTC m=+0.398065000,LastTimestamp:2026-10-30 22:06:16.644710766 +0000 UTC m=+0.398065000,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-77.ec2.internal,}"
Oct 30 22:06:16.660999 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:16.660981    2579 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Oct 30 22:06:16.661062 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:16.661027    2579 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-77.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Oct 30 22:06:16.669312 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.669160    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-rdntg"
Oct 30 22:06:16.679633 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.679604    2579 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 22:06:16.681951 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.681936    2579 manager.go:341] "Starting Device Plugin manager"
Oct 30 22:06:16.682061 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:16.681971    2579 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 22:06:16.682061 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.681982    2579 server.go:87] "Starting device plugin registration server"
Oct 30 22:06:16.682225 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.682213    2579 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 22:06:16.682278 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.682227    2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 22:06:16.682355 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.682339    2579 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 22:06:16.682426 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.682416    2579 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 22:06:16.682426 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.682424    2579 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 22:06:16.682920 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:16.682898    2579 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 22:06:16.682994 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:16.682943    2579 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-77.ec2.internal\" not found"
Oct 30 22:06:16.685951 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:16.685931    2579 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Oct 30 22:06:16.686004 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:16.685915    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-77.ec2.internal.18e36df53ad04d0b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-77.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-130-77.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-130-77.ec2.internal,},FirstTimestamp:2026-10-30 22:06:16.644734219 +0000 UTC m=+0.398088454,LastTimestamp:2026-10-30 22:06:16.644734219 +0000 UTC m=+0.398088454,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-77.ec2.internal,}"
Oct 30 22:06:16.753838 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.753808    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 22:06:16.755532 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.754995    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 22:06:16.755532 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.755014    2579 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 22:06:16.755532 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.755035    2579 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 22:06:16.755532 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:16.755082    2579 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 30 22:06:16.769694 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.769678    2579 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 22:06:16.782789 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.782771    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 22:06:16.783525 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.783505    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-77.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 22:06:16.783611 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.783528    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-77.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 22:06:16.783611 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.783537    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-77.ec2.internal" event="NodeHasSufficientPID"
Oct 30 22:06:16.783611 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.783559    2579 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-130-77.ec2.internal"
Oct 30 22:06:16.802090 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.802053    2579 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-130-77.ec2.internal"
Oct 30 22:06:16.802090 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:16.802073    2579 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-77.ec2.internal\": node \"ip-10-0-130-77.ec2.internal\" not found"
Oct 30 22:06:16.837385 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:16.837368    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-77.ec2.internal\" not found"
Oct 30 22:06:16.855487 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.855458    2579 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-77.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-77.ec2.internal"]
Oct 30 22:06:16.855538 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.855520    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 22:06:16.861742 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.861727    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-77.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 22:06:16.861797 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.861755    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-77.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 22:06:16.861797 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.861770    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-77.ec2.internal" event="NodeHasSufficientPID"
Oct 30 22:06:16.863708 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.863695    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 22:06:16.863857 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.863843    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-77.ec2.internal"
Oct 30 22:06:16.863894 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.863871    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 22:06:16.864384 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.864368    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-77.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 22:06:16.864465 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.864396    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-77.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 22:06:16.864465 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.864374    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-77.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 22:06:16.864465 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.864440    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-77.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 22:06:16.864465 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.864410    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-77.ec2.internal" event="NodeHasSufficientPID"
Oct 30 22:06:16.864465 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.864456    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-77.ec2.internal" event="NodeHasSufficientPID"
Oct 30 22:06:16.866386 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.866373    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-77.ec2.internal"
Oct 30 22:06:16.866463 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.866394    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 22:06:16.867122 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.867105    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-77.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 22:06:16.867198 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.867132    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-77.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 22:06:16.867198 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.867141    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-77.ec2.internal" event="NodeHasSufficientPID"
Oct 30 22:06:16.896939 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:16.896924    2579 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-77.ec2.internal\" not found" node="ip-10-0-130-77.ec2.internal"
Oct 30 22:06:16.900032 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:16.900015    2579 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-77.ec2.internal\" not found" node="ip-10-0-130-77.ec2.internal"
Oct 30 22:06:16.926983 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.926962    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/19ea3e9d10869a3d260d4dc45edca0a4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-77.ec2.internal\" (UID: \"19ea3e9d10869a3d260d4dc45edca0a4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-77.ec2.internal"
Oct 30 22:06:16.927081 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.926990    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/19ea3e9d10869a3d260d4dc45edca0a4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-77.ec2.internal\" (UID: \"19ea3e9d10869a3d260d4dc45edca0a4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-77.ec2.internal"
Oct 30 22:06:16.927081 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:16.927016    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1929b84fce10dee760acfa96faf62b49-config\") pod \"kube-apiserver-proxy-ip-10-0-130-77.ec2.internal\" (UID: \"1929b84fce10dee760acfa96faf62b49\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-77.ec2.internal"
Oct 30 22:06:16.937446 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:16.937426    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-77.ec2.internal\" not found"
Oct 30 22:06:17.027118 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:17.027101    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1929b84fce10dee760acfa96faf62b49-config\") pod \"kube-apiserver-proxy-ip-10-0-130-77.ec2.internal\" (UID: \"1929b84fce10dee760acfa96faf62b49\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-77.ec2.internal"
Oct 30 22:06:17.027192 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:17.027122    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/19ea3e9d10869a3d260d4dc45edca0a4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-77.ec2.internal\" (UID: \"19ea3e9d10869a3d260d4dc45edca0a4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-77.ec2.internal"
Oct 30 22:06:17.027192 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:17.027138    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/19ea3e9d10869a3d260d4dc45edca0a4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-77.ec2.internal\" (UID: \"19ea3e9d10869a3d260d4dc45edca0a4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-77.ec2.internal"
Oct 30 22:06:17.027192 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:17.027187    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/19ea3e9d10869a3d260d4dc45edca0a4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-77.ec2.internal\" (UID: \"19ea3e9d10869a3d260d4dc45edca0a4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-77.ec2.internal"
Oct 30 22:06:17.027293 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:17.027196    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1929b84fce10dee760acfa96faf62b49-config\") pod \"kube-apiserver-proxy-ip-10-0-130-77.ec2.internal\" (UID: \"1929b84fce10dee760acfa96faf62b49\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-77.ec2.internal"
Oct 30 22:06:17.027293 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:17.027212    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/19ea3e9d10869a3d260d4dc45edca0a4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-77.ec2.internal\" (UID: \"19ea3e9d10869a3d260d4dc45edca0a4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-77.ec2.internal"
Oct 30 22:06:17.038398 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:17.038381    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-77.ec2.internal\" not found"
Oct 30 22:06:17.139241 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:17.139196    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-77.ec2.internal\" not found"
Oct 30 22:06:17.200395 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:17.200367    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-77.ec2.internal"
Oct 30 22:06:17.203343 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:17.203325    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-77.ec2.internal"
Oct 30 22:06:17.239425 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:17.239403    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-77.ec2.internal\" not found"
Oct 30 22:06:17.340557 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:17.340532    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-77.ec2.internal\" not found"
Oct 30 22:06:17.441432 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:17.441355    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-77.ec2.internal\" not found"
Oct 30 22:06:17.541821 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:17.541784    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-77.ec2.internal\" not found"
Oct 30 22:06:17.560264 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:17.560220    2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 30 22:06:17.560412 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:17.560361    2579 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 22:06:17.641894 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:17.641867    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-77.ec2.internal\" not found"
Oct 30 22:06:17.674749 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:17.674713    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 22:01:16 +0000 UTC" deadline="2028-05-21 02:20:39.037485171 +0000 UTC"
Oct 30 22:06:17.674749 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:17.674747    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13636h14m21.362741531s"
Oct 30 22:06:17.696477 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:17.696382    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 22:06:17.697850 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:17.697827    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-77.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 22:06:17.697975 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:17.697870    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-77.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 22:06:17.697975 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:17.697887    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-77.ec2.internal" event="NodeHasSufficientPID"
Oct 30 22:06:17.742017 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:17.741989    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-77.ec2.internal\" not found"
Oct 30 22:06:17.804190 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:17.804160    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19ea3e9d10869a3d260d4dc45edca0a4.slice/crio-755aed24fd1307c4e0126a7576b17171fbcc4ec7d80432a009149af17d0d3049 WatchSource:0}: Error finding container 755aed24fd1307c4e0126a7576b17171fbcc4ec7d80432a009149af17d0d3049: Status 404 returned error can't find the container with id 755aed24fd1307c4e0126a7576b17171fbcc4ec7d80432a009149af17d0d3049
Oct 30 22:06:17.804436 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:17.804409    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1929b84fce10dee760acfa96faf62b49.slice/crio-da9614088a069dcc12ff6b8f4b299e40bbcfa0bf887bc17decc6d7f541d3723b WatchSource:0}: Error finding container da9614088a069dcc12ff6b8f4b299e40bbcfa0bf887bc17decc6d7f541d3723b: Status 404 returned error can't find the container with id da9614088a069dcc12ff6b8f4b299e40bbcfa0bf887bc17decc6d7f541d3723b
Oct 30 22:06:17.810976 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:17.810956    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 22:06:17.843126 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:17.843103    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-77.ec2.internal\" not found"
Oct 30 22:06:17.943954 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:17.943932    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-77.ec2.internal\" not found"
Oct 30 22:06:17.974826 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:17.974778    2579 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 22:06:18.044019 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:18.043993    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-77.ec2.internal\" not found"
Oct 30 22:06:18.086200 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.086173    2579 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 22:06:18.128365 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.128344    2579 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 22:06:18.224895 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.224814    2579 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-77.ec2.internal"
Oct 30 22:06:18.249770 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.249749    2579 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 22:06:18.250892 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.250870    2579 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-77.ec2.internal"
Oct 30 22:06:18.272964 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.272943    2579 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 22:06:18.598179 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.598113    2579 apiserver.go:52] "Watching apiserver"
Oct 30 22:06:18.624106 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.624084    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 30 22:06:18.629079 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.629058    2579 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 22:06:18.629198 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.629166    2579 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 22:06:18.629473 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.629454    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-4knxq","openshift-image-registry/node-ca-b74lk","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-77.ec2.internal","kube-system/konnectivity-agent-mm655","openshift-multus/multus-additional-cni-plugins-22c27","kube-system/global-pull-secret-syncer-rjtc8","kube-system/kube-apiserver-proxy-ip-10-0-130-77.ec2.internal","openshift-multus/multus-4s4vc","openshift-multus/network-metrics-daemon-6hhw9","openshift-network-diagnostics/network-check-target-vs5pt","openshift-network-operator/iptables-alerter-vw5fg","openshift-ovn-kubernetes/ovnkube-node-jcf8l","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bbkdz","openshift-cluster-node-tuning-operator/tuned-dr9c2"]
Oct 30 22:06:18.631968 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.631945    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vs5pt"
Oct 30 22:06:18.632074 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:18.632029    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vs5pt" podUID="dbf6b14c-5576-4ae3-bbb3-b3201aa7cb7c"
Oct 30 22:06:18.634364 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.634345    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bbkdz"
Oct 30 22:06:18.636593 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.636563    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dr9c2"
Oct 30 22:06:18.638776 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.638756    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mm655"
Oct 30 22:06:18.640976 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.640944    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-22c27"
Oct 30 22:06:18.643324 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.643302    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rjtc8"
Oct 30 22:06:18.643420 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:18.643384    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rjtc8" podUID="b8503b30-31d6-4454-a3fd-8bea4a527b04"
Oct 30 22:06:18.645343 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.645318    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6hhw9"
Oct 30 22:06:18.645438 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:18.645389    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6hhw9" podUID="fe4c45fd-690b-4111-9d15-93dc183b3e46"
Oct 30 22:06:18.647489 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.647469    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4s4vc"
Oct 30 22:06:18.650668 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.650623    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 22:06:18.650668 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.650655    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 22:06:18.650961 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.650944    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-b74lk"
Oct 30 22:06:18.651197 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.651179    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-k6z4j\""
Oct 30 22:06:18.651281 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.651215    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 22:06:18.652176 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.652148    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-g5ww4\""
Oct 30 22:06:18.652291 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.652278    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 22:06:18.653624 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.653585    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 22:06:18.654956 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.653914    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4knxq"
Oct 30 22:06:18.654956 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.654732    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-pwrc4\""
Oct 30 22:06:18.654956 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.654868    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 22:06:18.654956 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.654949    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 22:06:18.656742 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.656687    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vw5fg"
Oct 30 22:06:18.662423 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.659376    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l"
Oct 30 22:06:18.679328 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.679312    2579 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 22:06:18.679509 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.679471    2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-w5bdr"
Oct 30 22:06:18.688257 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.687943    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-w5bdr"
Oct 30 22:06:18.726372 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.726352    2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 22:06:18.735719 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.735691    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/015f218d-bf1f-4e7e-bb5d-b805af003246-node-log\") pod \"ovnkube-node-jcf8l\" (UID: \"015f218d-bf1f-4e7e-bb5d-b805af003246\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l"
Oct 30 22:06:18.735820 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.735753    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7b50e755-6b0b-4d45-b7d3-c822127e6cb1-etc-kubernetes\") pod \"multus-4s4vc\" (UID: \"7b50e755-6b0b-4d45-b7d3-c822127e6cb1\") " pod="openshift-multus/multus-4s4vc"
Oct 30 22:06:18.735820 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.735793    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/015f218d-bf1f-4e7e-bb5d-b805af003246-ovnkube-script-lib\") pod \"ovnkube-node-jcf8l\" (UID: \"015f218d-bf1f-4e7e-bb5d-b805af003246\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l"
Oct 30 22:06:18.735920 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.735845    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/87f77489-c825-4f2c-a6ce-70e06f68f187-tmp\") pod \"tuned-dr9c2\" (UID: \"87f77489-c825-4f2c-a6ce-70e06f68f187\") " pod="openshift-cluster-node-tuning-operator/tuned-dr9c2"
Oct 30 22:06:18.735920 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.735893    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/15e3a459-855f-4181-825c-345e9b2e62e1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-22c27\" (UID: \"15e3a459-855f-4181-825c-345e9b2e62e1\") " pod="openshift-multus/multus-additional-cni-plugins-22c27"
Oct 30 22:06:18.735999 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.735960    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7b50e755-6b0b-4d45-b7d3-c822127e6cb1-multus-cni-dir\") pod \"multus-4s4vc\" (UID: \"7b50e755-6b0b-4d45-b7d3-c822127e6cb1\") " pod="openshift-multus/multus-4s4vc"
Oct 30 22:06:18.736050 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.735997    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da9b75bf-8af2-450f-9a98-c265eea48282-tmp-dir\") pod \"node-resolver-4knxq\" (UID: \"da9b75bf-8af2-450f-9a98-c265eea48282\") " pod="openshift-dns/node-resolver-4knxq"
Oct 30 22:06:18.736050 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.736028    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8286n\" (UniqueName: \"kubernetes.io/projected/dbf6b14c-5576-4ae3-bbb3-b3201aa7cb7c-kube-api-access-8286n\") pod \"network-check-target-vs5pt\" (UID: \"dbf6b14c-5576-4ae3-bbb3-b3201aa7cb7c\") " pod="openshift-network-diagnostics/network-check-target-vs5pt"
Oct 30 22:06:18.736133 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.736067    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/015f218d-bf1f-4e7e-bb5d-b805af003246-host-run-netns\") pod \"ovnkube-node-jcf8l\" (UID: \"015f218d-bf1f-4e7e-bb5d-b805af003246\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l"
Oct 30 22:06:18.736167 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.736154    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/015f218d-bf1f-4e7e-bb5d-b805af003246-run-ovn\") pod \"ovnkube-node-jcf8l\" (UID: \"015f218d-bf1f-4e7e-bb5d-b805af003246\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l"
Oct 30 22:06:18.736207 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.736182    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/87f77489-c825-4f2c-a6ce-70e06f68f187-etc-sysconfig\") pod \"tuned-dr9c2\" (UID: \"87f77489-c825-4f2c-a6ce-70e06f68f187\") " pod="openshift-cluster-node-tuning-operator/tuned-dr9c2"
Oct 30 22:06:18.736240 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.736206    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/15e3a459-855f-4181-825c-345e9b2e62e1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-22c27\" (UID: \"15e3a459-855f-4181-825c-345e9b2e62e1\") " pod="openshift-multus/multus-additional-cni-plugins-22c27"
Oct 30 22:06:18.736286 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.736262    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ncxp\" (UniqueName: \"kubernetes.io/projected/15e3a459-855f-4181-825c-345e9b2e62e1-kube-api-access-4ncxp\") pod \"multus-additional-cni-plugins-22c27\" (UID: \"15e3a459-855f-4181-825c-345e9b2e62e1\") " pod="openshift-multus/multus-additional-cni-plugins-22c27"
Oct 30 22:06:18.736326 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.736297    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkshh\" (UniqueName: \"kubernetes.io/projected/da9b75bf-8af2-450f-9a98-c265eea48282-kube-api-access-mkshh\") pod \"node-resolver-4knxq\" (UID: \"da9b75bf-8af2-450f-9a98-c265eea48282\") " pod="openshift-dns/node-resolver-4knxq"
Oct 30 22:06:18.736366 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.736321    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2e4eaf5f-8a83-4681-b040-65676affdf7e-host-slash\") pod \"iptables-alerter-vw5fg\" (UID: \"2e4eaf5f-8a83-4681-b040-65676affdf7e\") " pod="openshift-network-operator/iptables-alerter-vw5fg"
Oct 30 22:06:18.736402 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.736365    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0e128ead-13e5-41f3-9ae7-6f944b5f18ee-etc-selinux\") pod \"aws-ebs-csi-driver-node-bbkdz\" (UID: \"0e128ead-13e5-41f3-9ae7-6f944b5f18ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bbkdz"
Oct 30 22:06:18.736402 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.736390    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7b50e755-6b0b-4d45-b7d3-c822127e6cb1-host-var-lib-cni-bin\") pod \"multus-4s4vc\" (UID: \"7b50e755-6b0b-4d45-b7d3-c822127e6cb1\") " pod="openshift-multus/multus-4s4vc"
Oct 30 22:06:18.736490 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.736415    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2e4eaf5f-8a83-4681-b040-65676affdf7e-iptables-alerter-script\") pod \"iptables-alerter-vw5fg\" (UID: \"2e4eaf5f-8a83-4681-b040-65676affdf7e\") " pod="openshift-network-operator/iptables-alerter-vw5fg"
Oct 30 22:06:18.736490 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.736473    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/87f77489-c825-4f2c-a6ce-70e06f68f187-etc-tuned\") pod \"tuned-dr9c2\" (UID: \"87f77489-c825-4f2c-a6ce-70e06f68f187\") " pod="openshift-cluster-node-tuning-operator/tuned-dr9c2"
Oct 30 22:06:18.736564 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.736511    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/15e3a459-855f-4181-825c-345e9b2e62e1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-22c27\" (UID: \"15e3a459-855f-4181-825c-345e9b2e62e1\") " pod="openshift-multus/multus-additional-cni-plugins-22c27"
Oct 30 22:06:18.736612 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.736556    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b8503b30-31d6-4454-a3fd-8bea4a527b04-dbus\") pod \"global-pull-secret-syncer-rjtc8\" (UID: \"b8503b30-31d6-4454-a3fd-8bea4a527b04\") " pod="kube-system/global-pull-secret-syncer-rjtc8"
Oct 30 22:06:18.736612 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.736601    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7b50e755-6b0b-4d45-b7d3-c822127e6cb1-cnibin\") pod \"multus-4s4vc\" (UID: \"7b50e755-6b0b-4d45-b7d3-c822127e6cb1\") " pod="openshift-multus/multus-4s4vc"
Oct 30 22:06:18.736684 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.736628    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7b50e755-6b0b-4d45-b7d3-c822127e6cb1-host-var-lib-cni-multus\") pod \"multus-4s4vc\" (UID: \"7b50e755-6b0b-4d45-b7d3-c822127e6cb1\") " pod="openshift-multus/multus-4s4vc"
Oct 30 22:06:18.736684 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.736644    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7b50e755-6b0b-4d45-b7d3-c822127e6cb1-multus-conf-dir\") pod \"multus-4s4vc\" (UID: \"7b50e755-6b0b-4d45-b7d3-c822127e6cb1\") " pod="openshift-multus/multus-4s4vc"
Oct 30 22:06:18.736684 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.736659    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/da9b75bf-8af2-450f-9a98-c265eea48282-hosts-file\") pod \"node-resolver-4knxq\" (UID: \"da9b75bf-8af2-450f-9a98-c265eea48282\") " pod="openshift-dns/node-resolver-4knxq"
Oct 30 22:06:18.736684 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.736675    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/015f218d-bf1f-4e7e-bb5d-b805af003246-etc-openvswitch\") pod \"ovnkube-node-jcf8l\" (UID: \"015f218d-bf1f-4e7e-bb5d-b805af003246\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l"
Oct 30 22:06:18.736817 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.736718    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mn4fl\" (UniqueName: \"kubernetes.io/projected/0e128ead-13e5-41f3-9ae7-6f944b5f18ee-kube-api-access-mn4fl\") pod \"aws-ebs-csi-driver-node-bbkdz\" (UID: \"0e128ead-13e5-41f3-9ae7-6f944b5f18ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bbkdz"
Oct 30 22:06:18.736817 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.736736    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/87f77489-c825-4f2c-a6ce-70e06f68f187-etc-sysctl-conf\") pod \"tuned-dr9c2\" (UID: \"87f77489-c825-4f2c-a6ce-70e06f68f187\") " pod="openshift-cluster-node-tuning-operator/tuned-dr9c2"
Oct 30 22:06:18.736817 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.736758    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/87f77489-c825-4f2c-a6ce-70e06f68f187-host\") pod \"tuned-dr9c2\" (UID: \"87f77489-c825-4f2c-a6ce-70e06f68f187\") " pod="openshift-cluster-node-tuning-operator/tuned-dr9c2"
Oct 30 22:06:18.736817 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.736785    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7b50e755-6b0b-4d45-b7d3-c822127e6cb1-host-run-multus-certs\") pod \"multus-4s4vc\" (UID: \"7b50e755-6b0b-4d45-b7d3-c822127e6cb1\") " pod="openshift-multus/multus-4s4vc"
Oct 30 22:06:18.736817 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.736800    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/015f218d-bf1f-4e7e-bb5d-b805af003246-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jcf8l\" (UID: \"015f218d-bf1f-4e7e-bb5d-b805af003246\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l"
Oct 30 22:06:18.736817 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.736816    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0e128ead-13e5-41f3-9ae7-6f944b5f18ee-registration-dir\") pod \"aws-ebs-csi-driver-node-bbkdz\" (UID: \"0e128ead-13e5-41f3-9ae7-6f944b5f18ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bbkdz"
Oct 30 22:06:18.737022 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.736832    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/15e3a459-855f-4181-825c-345e9b2e62e1-system-cni-dir\") pod \"multus-additional-cni-plugins-22c27\" (UID: \"15e3a459-855f-4181-825c-345e9b2e62e1\") " pod="openshift-multus/multus-additional-cni-plugins-22c27"
Oct 30 22:06:18.737022 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.736856    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7b50e755-6b0b-4d45-b7d3-c822127e6cb1-host-run-k8s-cni-cncf-io\") pod \"multus-4s4vc\" (UID: \"7b50e755-6b0b-4d45-b7d3-c822127e6cb1\") " pod="openshift-multus/multus-4s4vc"
Oct 30 22:06:18.737022 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.736900    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/015f218d-bf1f-4e7e-bb5d-b805af003246-host-cni-netd\") pod \"ovnkube-node-jcf8l\" (UID: \"015f218d-bf1f-4e7e-bb5d-b805af003246\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l"
Oct 30 22:06:18.737022 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.736918    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/015f218d-bf1f-4e7e-bb5d-b805af003246-env-overrides\") pod \"ovnkube-node-jcf8l\" (UID: \"015f218d-bf1f-4e7e-bb5d-b805af003246\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l"
Oct 30 22:06:18.737022 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.736946    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0e128ead-13e5-41f3-9ae7-6f944b5f18ee-socket-dir\") pod \"aws-ebs-csi-driver-node-bbkdz\" (UID: \"0e128ead-13e5-41f3-9ae7-6f944b5f18ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bbkdz"
Oct 30 22:06:18.737022 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.736962    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/87f77489-c825-4f2c-a6ce-70e06f68f187-etc-systemd\") pod \"tuned-dr9c2\" (UID: \"87f77489-c825-4f2c-a6ce-70e06f68f187\") " pod="openshift-cluster-node-tuning-operator/tuned-dr9c2"
Oct 30 22:06:18.737022 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.736993    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hc5rj\" (UniqueName: \"kubernetes.io/projected/87f77489-c825-4f2c-a6ce-70e06f68f187-kube-api-access-hc5rj\") pod \"tuned-dr9c2\" (UID: \"87f77489-c825-4f2c-a6ce-70e06f68f187\") " pod="openshift-cluster-node-tuning-operator/tuned-dr9c2"
Oct 30 22:06:18.737300 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.737017    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a3d96078-3ec8-40aa-855a-d1ba4acf2e74-agent-certs\") pod \"konnectivity-agent-mm655\" (UID: \"a3d96078-3ec8-40aa-855a-d1ba4acf2e74\") " pod="kube-system/konnectivity-agent-mm655"
Oct 30 22:06:18.737300 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.737073    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a3d96078-3ec8-40aa-855a-d1ba4acf2e74-konnectivity-ca\") pod \"konnectivity-agent-mm655\" (UID: \"a3d96078-3ec8-40aa-855a-d1ba4acf2e74\") " pod="kube-system/konnectivity-agent-mm655"
Oct 30 22:06:18.737300 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.737092    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/015f218d-bf1f-4e7e-bb5d-b805af003246-ovn-node-metrics-cert\") pod \"ovnkube-node-jcf8l\" (UID: \"015f218d-bf1f-4e7e-bb5d-b805af003246\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l"
Oct 30 22:06:18.737300 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.737120    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/15e3a459-855f-4181-825c-345e9b2e62e1-cnibin\") pod \"multus-additional-cni-plugins-22c27\" (UID: \"15e3a459-855f-4181-825c-345e9b2e62e1\") " pod="openshift-multus/multus-additional-cni-plugins-22c27"
Oct 30 22:06:18.737300 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.737135    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7b50e755-6b0b-4d45-b7d3-c822127e6cb1-multus-socket-dir-parent\") pod \"multus-4s4vc\" (UID: \"7b50e755-6b0b-4d45-b7d3-c822127e6cb1\") " pod="openshift-multus/multus-4s4vc"
Oct 30 22:06:18.737300 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.737152    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfxls\" (UniqueName: \"kubernetes.io/projected/2e4eaf5f-8a83-4681-b040-65676affdf7e-kube-api-access-tfxls\") pod \"iptables-alerter-vw5fg\" (UID: \"2e4eaf5f-8a83-4681-b040-65676affdf7e\") " pod="openshift-network-operator/iptables-alerter-vw5fg"
Oct 30 22:06:18.737300 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.737175    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe4c45fd-690b-4111-9d15-93dc183b3e46-metrics-certs\") pod \"network-metrics-daemon-6hhw9\" (UID: \"fe4c45fd-690b-4111-9d15-93dc183b3e46\") " pod="openshift-multus/network-metrics-daemon-6hhw9"
Oct 30 22:06:18.737300 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.737191    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/015f218d-bf1f-4e7e-bb5d-b805af003246-host-kubelet\") pod \"ovnkube-node-jcf8l\" (UID: \"015f218d-bf1f-4e7e-bb5d-b805af003246\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l"
Oct 30 22:06:18.737300 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.737210    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/015f218d-bf1f-4e7e-bb5d-b805af003246-host-slash\") pod \"ovnkube-node-jcf8l\" (UID: \"015f218d-bf1f-4e7e-bb5d-b805af003246\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l"
Oct 30 22:06:18.737300 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.737223    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0e128ead-13e5-41f3-9ae7-6f944b5f18ee-sys-fs\") pod \"aws-ebs-csi-driver-node-bbkdz\" (UID: \"0e128ead-13e5-41f3-9ae7-6f944b5f18ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bbkdz"
Oct 30 22:06:18.737300 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.737243    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/87f77489-c825-4f2c-a6ce-70e06f68f187-etc-modprobe-d\") pod \"tuned-dr9c2\" (UID: \"87f77489-c825-4f2c-a6ce-70e06f68f187\") " pod="openshift-cluster-node-tuning-operator/tuned-dr9c2"
Oct 30 22:06:18.737730 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.737355    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/87f77489-c825-4f2c-a6ce-70e06f68f187-sys\") pod \"tuned-dr9c2\" (UID: \"87f77489-c825-4f2c-a6ce-70e06f68f187\") " pod="openshift-cluster-node-tuning-operator/tuned-dr9c2"
Oct 30 22:06:18.737730 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.737376    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7b50e755-6b0b-4d45-b7d3-c822127e6cb1-cni-binary-copy\") pod \"multus-4s4vc\" (UID: \"7b50e755-6b0b-4d45-b7d3-c822127e6cb1\") " pod="openshift-multus/multus-4s4vc"
Oct 30 22:06:18.737730 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.737393    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/015f218d-bf1f-4e7e-bb5d-b805af003246-systemd-units\") pod \"ovnkube-node-jcf8l\" (UID: \"015f218d-bf1f-4e7e-bb5d-b805af003246\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l"
Oct 30 22:06:18.737730 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.737420    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b8503b30-31d6-4454-a3fd-8bea4a527b04-kubelet-config\") pod \"global-pull-secret-syncer-rjtc8\" (UID: \"b8503b30-31d6-4454-a3fd-8bea4a527b04\") " pod="kube-system/global-pull-secret-syncer-rjtc8"
Oct 30 22:06:18.737730 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.737439    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7b50e755-6b0b-4d45-b7d3-c822127e6cb1-host-run-netns\") pod \"multus-4s4vc\" (UID: \"7b50e755-6b0b-4d45-b7d3-c822127e6cb1\") " pod="openshift-multus/multus-4s4vc"
Oct 30 22:06:18.737730 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.737454    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7b50e755-6b0b-4d45-b7d3-c822127e6cb1-host-var-lib-kubelet\") pod \"multus-4s4vc\" (UID: \"7b50e755-6b0b-4d45-b7d3-c822127e6cb1\") " pod="openshift-multus/multus-4s4vc"
Oct 30 22:06:18.737730 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.737468    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7b50e755-6b0b-4d45-b7d3-c822127e6cb1-hostroot\") pod \"multus-4s4vc\" (UID: \"7b50e755-6b0b-4d45-b7d3-c822127e6cb1\") " pod="openshift-multus/multus-4s4vc"
Oct 30 22:06:18.737730 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.737493    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5d9bb\" (UniqueName: \"kubernetes.io/projected/7b50e755-6b0b-4d45-b7d3-c822127e6cb1-kube-api-access-5d9bb\") pod \"multus-4s4vc\" (UID: \"7b50e755-6b0b-4d45-b7d3-c822127e6cb1\") " pod="openshift-multus/multus-4s4vc"
Oct 30 22:06:18.737730 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.737511    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/015f218d-bf1f-4e7e-bb5d-b805af003246-var-lib-openvswitch\") pod \"ovnkube-node-jcf8l\" (UID: \"015f218d-bf1f-4e7e-bb5d-b805af003246\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l"
Oct 30 22:06:18.737730 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.737527    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/015f218d-bf1f-4e7e-bb5d-b805af003246-run-openvswitch\") pod \"ovnkube-node-jcf8l\" (UID: \"015f218d-bf1f-4e7e-bb5d-b805af003246\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l"
Oct 30 22:06:18.737730 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.737542    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/015f218d-bf1f-4e7e-bb5d-b805af003246-host-run-ovn-kubernetes\") pod \"ovnkube-node-jcf8l\" (UID: \"015f218d-bf1f-4e7e-bb5d-b805af003246\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l"
Oct 30 22:06:18.737730 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.737558    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/015f218d-bf1f-4e7e-bb5d-b805af003246-ovnkube-config\") pod \"ovnkube-node-jcf8l\" (UID: \"015f218d-bf1f-4e7e-bb5d-b805af003246\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l"
Oct 30 22:06:18.737730 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.737580    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/87f77489-c825-4f2c-a6ce-70e06f68f187-run\") pod \"tuned-dr9c2\" (UID: \"87f77489-c825-4f2c-a6ce-70e06f68f187\") " pod="openshift-cluster-node-tuning-operator/tuned-dr9c2"
Oct 30 22:06:18.737730 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.737602    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/15e3a459-855f-4181-825c-345e9b2e62e1-cni-binary-copy\") pod \"multus-additional-cni-plugins-22c27\" (UID: \"15e3a459-855f-4181-825c-345e9b2e62e1\") " pod="openshift-multus/multus-additional-cni-plugins-22c27"
Oct 30 22:06:18.737730 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.737628    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7b50e755-6b0b-4d45-b7d3-c822127e6cb1-system-cni-dir\") pod \"multus-4s4vc\" (UID: \"7b50e755-6b0b-4d45-b7d3-c822127e6cb1\") " pod="openshift-multus/multus-4s4vc"
Oct 30 22:06:18.737730 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.737645    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e9c20fdf-bea5-4d6c-b252-b29b08bfdf63-serviceca\") pod \"node-ca-b74lk\" (UID: \"e9c20fdf-bea5-4d6c-b252-b29b08bfdf63\") " pod="openshift-image-registry/node-ca-b74lk"
Oct 30 22:06:18.737730 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.737660    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57q7f\" (UniqueName: \"kubernetes.io/projected/015f218d-bf1f-4e7e-bb5d-b805af003246-kube-api-access-57q7f\") pod \"ovnkube-node-jcf8l\" (UID: \"015f218d-bf1f-4e7e-bb5d-b805af003246\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l"
Oct 30 22:06:18.738442 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.737675    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0e128ead-13e5-41f3-9ae7-6f944b5f18ee-device-dir\") pod \"aws-ebs-csi-driver-node-bbkdz\" (UID: \"0e128ead-13e5-41f3-9ae7-6f944b5f18ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bbkdz"
Oct 30 22:06:18.738442 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.737688    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/87f77489-c825-4f2c-a6ce-70e06f68f187-etc-kubernetes\") pod \"tuned-dr9c2\" (UID: \"87f77489-c825-4f2c-a6ce-70e06f68f187\") " pod="openshift-cluster-node-tuning-operator/tuned-dr9c2"
Oct 30 22:06:18.738442 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.737702    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/87f77489-c825-4f2c-a6ce-70e06f68f187-var-lib-kubelet\") pod \"tuned-dr9c2\" (UID: \"87f77489-c825-4f2c-a6ce-70e06f68f187\") " pod="openshift-cluster-node-tuning-operator/tuned-dr9c2"
Oct 30 22:06:18.738442 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.737729    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b8503b30-31d6-4454-a3fd-8bea4a527b04-original-pull-secret\") pod \"global-pull-secret-syncer-rjtc8\" (UID: \"b8503b30-31d6-4454-a3fd-8bea4a527b04\") " pod="kube-system/global-pull-secret-syncer-rjtc8"
Oct 30 22:06:18.738442 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.737749    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frxsh\" (UniqueName: \"kubernetes.io/projected/fe4c45fd-690b-4111-9d15-93dc183b3e46-kube-api-access-frxsh\") pod \"network-metrics-daemon-6hhw9\" (UID: \"fe4c45fd-690b-4111-9d15-93dc183b3e46\") " pod="openshift-multus/network-metrics-daemon-6hhw9"
Oct 30 22:06:18.738442 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.737764    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e9c20fdf-bea5-4d6c-b252-b29b08bfdf63-host\") pod \"node-ca-b74lk\" (UID: \"e9c20fdf-bea5-4d6c-b252-b29b08bfdf63\") " pod="openshift-image-registry/node-ca-b74lk"
Oct 30 22:06:18.738442 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.737777    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/015f218d-bf1f-4e7e-bb5d-b805af003246-run-systemd\") pod \"ovnkube-node-jcf8l\" (UID: \"015f218d-bf1f-4e7e-bb5d-b805af003246\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l"
Oct 30 22:06:18.738442 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.737792    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/015f218d-bf1f-4e7e-bb5d-b805af003246-host-cni-bin\") pod \"ovnkube-node-jcf8l\" (UID: \"015f218d-bf1f-4e7e-bb5d-b805af003246\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l"
Oct 30 22:06:18.738442 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.737818    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/87f77489-c825-4f2c-a6ce-70e06f68f187-etc-sysctl-d\") pod \"tuned-dr9c2\" (UID: \"87f77489-c825-4f2c-a6ce-70e06f68f187\") " pod="openshift-cluster-node-tuning-operator/tuned-dr9c2"
Oct 30 22:06:18.738442 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.737843    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7b50e755-6b0b-4d45-b7d3-c822127e6cb1-multus-daemon-config\") pod \"multus-4s4vc\" (UID: \"7b50e755-6b0b-4d45-b7d3-c822127e6cb1\") " pod="openshift-multus/multus-4s4vc"
Oct 30 22:06:18.738442 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.737868    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cdcp\" (UniqueName: \"kubernetes.io/projected/e9c20fdf-bea5-4d6c-b252-b29b08bfdf63-kube-api-access-5cdcp\") pod \"node-ca-b74lk\" (UID: \"e9c20fdf-bea5-4d6c-b252-b29b08bfdf63\") " pod="openshift-image-registry/node-ca-b74lk"
Oct 30 22:06:18.738442 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.738017    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/015f218d-bf1f-4e7e-bb5d-b805af003246-log-socket\") pod \"ovnkube-node-jcf8l\" (UID: \"015f218d-bf1f-4e7e-bb5d-b805af003246\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l"
Oct 30 22:06:18.738442 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.738034    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0e128ead-13e5-41f3-9ae7-6f944b5f18ee-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bbkdz\" (UID: \"0e128ead-13e5-41f3-9ae7-6f944b5f18ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bbkdz"
Oct 30 22:06:18.738442 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.738064    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/87f77489-c825-4f2c-a6ce-70e06f68f187-lib-modules\") pod \"tuned-dr9c2\" (UID: \"87f77489-c825-4f2c-a6ce-70e06f68f187\") " pod="openshift-cluster-node-tuning-operator/tuned-dr9c2"
Oct 30 22:06:18.751463 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.751443    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-44l7l\""
Oct 30 22:06:18.751569 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.751476    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 22:06:18.751569 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.751512    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 22:06:18.751682 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.751646    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 22:06:18.751738 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.751683    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 22:06:18.752004 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.751953    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 22:06:18.752157 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.752138    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-7kw8g\""
Oct 30 22:06:18.753175 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.752289    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-8wxtm\""
Oct 30 22:06:18.753175 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.752310    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 22:06:18.753175 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.752366    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 22:06:18.753175 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.752442    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 22:06:18.753175 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.752486    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 22:06:18.753175 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.752517    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 22:06:18.753175 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.752542    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 22:06:18.753175 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.752570    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 22:06:18.753175 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.752592    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 22:06:18.753175 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.752613    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 22:06:18.753175 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.752664    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 22:06:18.753175 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.752702    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-tjmjw\""
Oct 30 22:06:18.753175 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.752754    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-nn9g8\""
Oct 30 22:06:18.753175 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.752778    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 22:06:18.753175 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.752848    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-xwfpf\""
Oct 30 22:06:18.753175 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.752929    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 22:06:18.753175 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.753029    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 22:06:18.753175 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.753148    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 22:06:18.753175 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.753153    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 22:06:18.760225 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.760177    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-77.ec2.internal" event={"ID":"1929b84fce10dee760acfa96faf62b49","Type":"ContainerStarted","Data":"da9614088a069dcc12ff6b8f4b299e40bbcfa0bf887bc17decc6d7f541d3723b"}
Oct 30 22:06:18.761514 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.761471    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-77.ec2.internal" event={"ID":"19ea3e9d10869a3d260d4dc45edca0a4","Type":"ContainerStarted","Data":"755aed24fd1307c4e0126a7576b17171fbcc4ec7d80432a009149af17d0d3049"}
Oct 30 22:06:18.838563 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.838528    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/87f77489-c825-4f2c-a6ce-70e06f68f187-etc-sysctl-conf\") pod \"tuned-dr9c2\" (UID: \"87f77489-c825-4f2c-a6ce-70e06f68f187\") " pod="openshift-cluster-node-tuning-operator/tuned-dr9c2"
Oct 30 22:06:18.838563 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.838565    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/87f77489-c825-4f2c-a6ce-70e06f68f187-host\") pod \"tuned-dr9c2\" (UID: \"87f77489-c825-4f2c-a6ce-70e06f68f187\") " pod="openshift-cluster-node-tuning-operator/tuned-dr9c2"
Oct 30 22:06:18.838742 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.838590    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7b50e755-6b0b-4d45-b7d3-c822127e6cb1-host-run-multus-certs\") pod \"multus-4s4vc\" (UID: \"7b50e755-6b0b-4d45-b7d3-c822127e6cb1\") " pod="openshift-multus/multus-4s4vc"
Oct 30 22:06:18.838742 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.838615    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/015f218d-bf1f-4e7e-bb5d-b805af003246-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jcf8l\" (UID: \"015f218d-bf1f-4e7e-bb5d-b805af003246\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l"
Oct 30 22:06:18.838742 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.838642    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0e128ead-13e5-41f3-9ae7-6f944b5f18ee-registration-dir\") pod \"aws-ebs-csi-driver-node-bbkdz\" (UID: \"0e128ead-13e5-41f3-9ae7-6f944b5f18ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bbkdz"
Oct 30 22:06:18.838742 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.838669    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/15e3a459-855f-4181-825c-345e9b2e62e1-system-cni-dir\") pod \"multus-additional-cni-plugins-22c27\" (UID: \"15e3a459-855f-4181-825c-345e9b2e62e1\") " pod="openshift-multus/multus-additional-cni-plugins-22c27"
Oct 30 22:06:18.838742 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.838693    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7b50e755-6b0b-4d45-b7d3-c822127e6cb1-host-run-k8s-cni-cncf-io\") pod \"multus-4s4vc\" (UID: \"7b50e755-6b0b-4d45-b7d3-c822127e6cb1\") " pod="openshift-multus/multus-4s4vc"
Oct 30 22:06:18.838742 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.838715    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/015f218d-bf1f-4e7e-bb5d-b805af003246-host-cni-netd\") pod \"ovnkube-node-jcf8l\" (UID: \"015f218d-bf1f-4e7e-bb5d-b805af003246\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l"
Oct 30 22:06:18.838742 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.838738    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/015f218d-bf1f-4e7e-bb5d-b805af003246-env-overrides\") pod \"ovnkube-node-jcf8l\" (UID: \"015f218d-bf1f-4e7e-bb5d-b805af003246\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l"
Oct 30 22:06:18.839032 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.838761    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/87f77489-c825-4f2c-a6ce-70e06f68f187-etc-sysctl-conf\") pod \"tuned-dr9c2\" (UID: \"87f77489-c825-4f2c-a6ce-70e06f68f187\") " pod="openshift-cluster-node-tuning-operator/tuned-dr9c2"
Oct 30 22:06:18.839032 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.838765    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0e128ead-13e5-41f3-9ae7-6f944b5f18ee-socket-dir\") pod \"aws-ebs-csi-driver-node-bbkdz\" (UID: \"0e128ead-13e5-41f3-9ae7-6f944b5f18ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bbkdz"
Oct 30 22:06:18.839032 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.838802    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/87f77489-c825-4f2c-a6ce-70e06f68f187-etc-systemd\") pod \"tuned-dr9c2\" (UID: \"87f77489-c825-4f2c-a6ce-70e06f68f187\") " pod="openshift-cluster-node-tuning-operator/tuned-dr9c2"
Oct 30 22:06:18.839032 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.838821    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hc5rj\" (UniqueName: \"kubernetes.io/projected/87f77489-c825-4f2c-a6ce-70e06f68f187-kube-api-access-hc5rj\") pod \"tuned-dr9c2\" (UID: \"87f77489-c825-4f2c-a6ce-70e06f68f187\") " pod="openshift-cluster-node-tuning-operator/tuned-dr9c2"
Oct 30 22:06:18.839032 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.838852    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a3d96078-3ec8-40aa-855a-d1ba4acf2e74-agent-certs\") pod \"konnectivity-agent-mm655\" (UID: \"a3d96078-3ec8-40aa-855a-d1ba4acf2e74\") " pod="kube-system/konnectivity-agent-mm655"
Oct 30 22:06:18.839032 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.838871    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a3d96078-3ec8-40aa-855a-d1ba4acf2e74-konnectivity-ca\") pod \"konnectivity-agent-mm655\" (UID: \"a3d96078-3ec8-40aa-855a-d1ba4acf2e74\") " pod="kube-system/konnectivity-agent-mm655"
Oct 30 22:06:18.839032 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.838891    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/015f218d-bf1f-4e7e-bb5d-b805af003246-ovn-node-metrics-cert\") pod \"ovnkube-node-jcf8l\" (UID: \"015f218d-bf1f-4e7e-bb5d-b805af003246\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l"
Oct 30 22:06:18.839032 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.838907    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/15e3a459-855f-4181-825c-345e9b2e62e1-cnibin\") pod \"multus-additional-cni-plugins-22c27\" (UID: \"15e3a459-855f-4181-825c-345e9b2e62e1\") " pod="openshift-multus/multus-additional-cni-plugins-22c27"
Oct 30 22:06:18.839032 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.838905    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0e128ead-13e5-41f3-9ae7-6f944b5f18ee-socket-dir\") pod \"aws-ebs-csi-driver-node-bbkdz\" (UID: \"0e128ead-13e5-41f3-9ae7-6f944b5f18ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bbkdz"
Oct 30 22:06:18.839032 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.838933    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7b50e755-6b0b-4d45-b7d3-c822127e6cb1-multus-socket-dir-parent\") pod \"multus-4s4vc\" (UID: \"7b50e755-6b0b-4d45-b7d3-c822127e6cb1\") " pod="openshift-multus/multus-4s4vc"
Oct 30 22:06:18.839032 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.838972    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfxls\" (UniqueName: \"kubernetes.io/projected/2e4eaf5f-8a83-4681-b040-65676affdf7e-kube-api-access-tfxls\") pod \"iptables-alerter-vw5fg\" (UID: \"2e4eaf5f-8a83-4681-b040-65676affdf7e\") " pod="openshift-network-operator/iptables-alerter-vw5fg"
Oct 30 22:06:18.839032 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.838998    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe4c45fd-690b-4111-9d15-93dc183b3e46-metrics-certs\") pod \"network-metrics-daemon-6hhw9\" (UID: \"fe4c45fd-690b-4111-9d15-93dc183b3e46\") " pod="openshift-multus/network-metrics-daemon-6hhw9"
Oct 30 22:06:18.839032 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.839020    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/015f218d-bf1f-4e7e-bb5d-b805af003246-host-kubelet\") pod \"ovnkube-node-jcf8l\" (UID: \"015f218d-bf1f-4e7e-bb5d-b805af003246\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l"
Oct 30 22:06:18.839585 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.839058    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/015f218d-bf1f-4e7e-bb5d-b805af003246-host-slash\") pod \"ovnkube-node-jcf8l\" (UID: \"015f218d-bf1f-4e7e-bb5d-b805af003246\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l"
Oct 30 22:06:18.839585 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.839080    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0e128ead-13e5-41f3-9ae7-6f944b5f18ee-sys-fs\") pod \"aws-ebs-csi-driver-node-bbkdz\" (UID: \"0e128ead-13e5-41f3-9ae7-6f944b5f18ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bbkdz"
Oct 30 22:06:18.839585 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.839079    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0e128ead-13e5-41f3-9ae7-6f944b5f18ee-registration-dir\") pod \"aws-ebs-csi-driver-node-bbkdz\" (UID: \"0e128ead-13e5-41f3-9ae7-6f944b5f18ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bbkdz"
Oct 30 22:06:18.839585 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.839123    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/15e3a459-855f-4181-825c-345e9b2e62e1-system-cni-dir\") pod \"multus-additional-cni-plugins-22c27\" (UID: \"15e3a459-855f-4181-825c-345e9b2e62e1\") " pod="openshift-multus/multus-additional-cni-plugins-22c27"
Oct 30 22:06:18.839585 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.839153    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/87f77489-c825-4f2c-a6ce-70e06f68f187-etc-modprobe-d\") pod \"tuned-dr9c2\" (UID: \"87f77489-c825-4f2c-a6ce-70e06f68f187\") " pod="openshift-cluster-node-tuning-operator/tuned-dr9c2"
Oct 30 22:06:18.839585 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.839156    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/87f77489-c825-4f2c-a6ce-70e06f68f187-etc-systemd\") pod \"tuned-dr9c2\" (UID: \"87f77489-c825-4f2c-a6ce-70e06f68f187\") " pod="openshift-cluster-node-tuning-operator/tuned-dr9c2"
Oct 30 22:06:18.839585 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.839177    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/87f77489-c825-4f2c-a6ce-70e06f68f187-sys\") pod \"tuned-dr9c2\" (UID: \"87f77489-c825-4f2c-a6ce-70e06f68f187\") " pod="openshift-cluster-node-tuning-operator/tuned-dr9c2"
Oct 30 22:06:18.839585 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.839203    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7b50e755-6b0b-4d45-b7d3-c822127e6cb1-cni-binary-copy\") pod \"multus-4s4vc\" (UID: \"7b50e755-6b0b-4d45-b7d3-c822127e6cb1\") " pod="openshift-multus/multus-4s4vc"
Oct 30 22:06:18.839585 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.839248    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/015f218d-bf1f-4e7e-bb5d-b805af003246-systemd-units\") pod \"ovnkube-node-jcf8l\" (UID: \"015f218d-bf1f-4e7e-bb5d-b805af003246\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l"
Oct 30 22:06:18.839585 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.839276    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b8503b30-31d6-4454-a3fd-8bea4a527b04-kubelet-config\") pod \"global-pull-secret-syncer-rjtc8\" (UID: \"b8503b30-31d6-4454-a3fd-8bea4a527b04\") " pod="kube-system/global-pull-secret-syncer-rjtc8"
Oct 30 22:06:18.839585 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.839300    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7b50e755-6b0b-4d45-b7d3-c822127e6cb1-host-run-netns\") pod \"multus-4s4vc\" (UID: \"7b50e755-6b0b-4d45-b7d3-c822127e6cb1\") " pod="openshift-multus/multus-4s4vc"
Oct 30 22:06:18.839585 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.839330    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7b50e755-6b0b-4d45-b7d3-c822127e6cb1-host-var-lib-kubelet\") pod \"multus-4s4vc\" (UID: \"7b50e755-6b0b-4d45-b7d3-c822127e6cb1\") " pod="openshift-multus/multus-4s4vc"
Oct 30 22:06:18.839585 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.839349    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7b50e755-6b0b-4d45-b7d3-c822127e6cb1-hostroot\") pod \"multus-4s4vc\" (UID: \"7b50e755-6b0b-4d45-b7d3-c822127e6cb1\") " pod="openshift-multus/multus-4s4vc"
Oct 30 22:06:18.839585 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.839367    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5d9bb\" (UniqueName: \"kubernetes.io/projected/7b50e755-6b0b-4d45-b7d3-c822127e6cb1-kube-api-access-5d9bb\") pod \"multus-4s4vc\" (UID: \"7b50e755-6b0b-4d45-b7d3-c822127e6cb1\") " pod="openshift-multus/multus-4s4vc"
Oct 30 22:06:18.839585 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.839390    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/015f218d-bf1f-4e7e-bb5d-b805af003246-var-lib-openvswitch\") pod \"ovnkube-node-jcf8l\" (UID: \"015f218d-bf1f-4e7e-bb5d-b805af003246\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l"
Oct 30 22:06:18.839585 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.839412    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/015f218d-bf1f-4e7e-bb5d-b805af003246-run-openvswitch\") pod \"ovnkube-node-jcf8l\" (UID: \"015f218d-bf1f-4e7e-bb5d-b805af003246\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l"
Oct 30 22:06:18.839585 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.839439    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/015f218d-bf1f-4e7e-bb5d-b805af003246-host-run-ovn-kubernetes\") pod \"ovnkube-node-jcf8l\" (UID: \"015f218d-bf1f-4e7e-bb5d-b805af003246\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l"
Oct 30 22:06:18.840401 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.839454    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7b50e755-6b0b-4d45-b7d3-c822127e6cb1-host-run-k8s-cni-cncf-io\") pod \"multus-4s4vc\" (UID: \"7b50e755-6b0b-4d45-b7d3-c822127e6cb1\") " pod="openshift-multus/multus-4s4vc"
Oct 30 22:06:18.840401 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.839539    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b8503b30-31d6-4454-a3fd-8bea4a527b04-kubelet-config\") pod \"global-pull-secret-syncer-rjtc8\" (UID: \"b8503b30-31d6-4454-a3fd-8bea4a527b04\") " pod="kube-system/global-pull-secret-syncer-rjtc8"
Oct 30 22:06:18.840401 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.839596    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7b50e755-6b0b-4d45-b7d3-c822127e6cb1-host-run-netns\") pod \"multus-4s4vc\" (UID: \"7b50e755-6b0b-4d45-b7d3-c822127e6cb1\") " pod="openshift-multus/multus-4s4vc"
Oct 30 22:06:18.840401 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.839629    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7b50e755-6b0b-4d45-b7d3-c822127e6cb1-host-var-lib-kubelet\") pod \"multus-4s4vc\" (UID: \"7b50e755-6b0b-4d45-b7d3-c822127e6cb1\") " pod="openshift-multus/multus-4s4vc"
Oct 30 22:06:18.840401 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.839670    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7b50e755-6b0b-4d45-b7d3-c822127e6cb1-hostroot\") pod \"multus-4s4vc\" (UID: \"7b50e755-6b0b-4d45-b7d3-c822127e6cb1\") " pod="openshift-multus/multus-4s4vc"
Oct 30 22:06:18.840401 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.839712    2579 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 22:06:18.840401 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.839895    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/015f218d-bf1f-4e7e-bb5d-b805af003246-var-lib-openvswitch\") pod \"ovnkube-node-jcf8l\" (UID: \"015f218d-bf1f-4e7e-bb5d-b805af003246\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l"
Oct 30 22:06:18.840401 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.839926    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/015f218d-bf1f-4e7e-bb5d-b805af003246-run-openvswitch\") pod \"ovnkube-node-jcf8l\" (UID: \"015f218d-bf1f-4e7e-bb5d-b805af003246\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l"
Oct 30 22:06:18.840401 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.839954    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/015f218d-bf1f-4e7e-bb5d-b805af003246-host-run-ovn-kubernetes\") pod \"ovnkube-node-jcf8l\" (UID: \"015f218d-bf1f-4e7e-bb5d-b805af003246\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l"
Oct 30 22:06:18.840401 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.838976    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/87f77489-c825-4f2c-a6ce-70e06f68f187-host\") pod \"tuned-dr9c2\" (UID: \"87f77489-c825-4f2c-a6ce-70e06f68f187\") " pod="openshift-cluster-node-tuning-operator/tuned-dr9c2"
Oct 30 22:06:18.840401 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.840022    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/15e3a459-855f-4181-825c-345e9b2e62e1-cnibin\") pod \"multus-additional-cni-plugins-22c27\" (UID: \"15e3a459-855f-4181-825c-345e9b2e62e1\") " pod="openshift-multus/multus-additional-cni-plugins-22c27"
Oct 30 22:06:18.840401 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.840060    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/015f218d-bf1f-4e7e-bb5d-b805af003246-ovnkube-config\") pod \"ovnkube-node-jcf8l\" (UID: \"015f218d-bf1f-4e7e-bb5d-b805af003246\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l"
Oct 30 22:06:18.840401 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.840146    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/015f218d-bf1f-4e7e-bb5d-b805af003246-systemd-units\") pod \"ovnkube-node-jcf8l\" (UID: \"015f218d-bf1f-4e7e-bb5d-b805af003246\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l"
Oct 30 22:06:18.840401 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.840158    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7b50e755-6b0b-4d45-b7d3-c822127e6cb1-multus-socket-dir-parent\") pod \"multus-4s4vc\" (UID: \"7b50e755-6b0b-4d45-b7d3-c822127e6cb1\") " pod="openshift-multus/multus-4s4vc"
Oct 30 22:06:18.840401 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.840216    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/015f218d-bf1f-4e7e-bb5d-b805af003246-host-cni-netd\") pod \"ovnkube-node-jcf8l\" (UID: \"015f218d-bf1f-4e7e-bb5d-b805af003246\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l"
Oct 30 22:06:18.840401 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.840352    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/015f218d-bf1f-4e7e-bb5d-b805af003246-host-kubelet\") pod \"ovnkube-node-jcf8l\" (UID: \"015f218d-bf1f-4e7e-bb5d-b805af003246\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l"
Oct 30 22:06:18.840401 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.840383    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a3d96078-3ec8-40aa-855a-d1ba4acf2e74-konnectivity-ca\") pod \"konnectivity-agent-mm655\" (UID: \"a3d96078-3ec8-40aa-855a-d1ba4acf2e74\") " pod="kube-system/konnectivity-agent-mm655"
Oct 30 22:06:18.840401 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.840391    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7b50e755-6b0b-4d45-b7d3-c822127e6cb1-host-run-multus-certs\") pod \"multus-4s4vc\" (UID: \"7b50e755-6b0b-4d45-b7d3-c822127e6cb1\") " pod="openshift-multus/multus-4s4vc"
Oct 30 22:06:18.841251 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:18.840458    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 22:06:18.841251 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:18.840506    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe4c45fd-690b-4111-9d15-93dc183b3e46-metrics-certs podName:fe4c45fd-690b-4111-9d15-93dc183b3e46 nodeName:}" failed. No retries permitted until 2026-10-30 22:06:19.340491623 +0000 UTC m=+3.093845854 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fe4c45fd-690b-4111-9d15-93dc183b3e46-metrics-certs") pod "network-metrics-daemon-6hhw9" (UID: "fe4c45fd-690b-4111-9d15-93dc183b3e46") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 22:06:18.841251 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.839019    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/015f218d-bf1f-4e7e-bb5d-b805af003246-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jcf8l\" (UID: \"015f218d-bf1f-4e7e-bb5d-b805af003246\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l"
Oct 30 22:06:18.841251 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.840599    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/015f218d-bf1f-4e7e-bb5d-b805af003246-env-overrides\") pod \"ovnkube-node-jcf8l\" (UID: \"015f218d-bf1f-4e7e-bb5d-b805af003246\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l"
Oct 30 22:06:18.841251 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.840644    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/015f218d-bf1f-4e7e-bb5d-b805af003246-host-slash\") pod \"ovnkube-node-jcf8l\" (UID: \"015f218d-bf1f-4e7e-bb5d-b805af003246\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l"
Oct 30 22:06:18.841251 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.840678    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/87f77489-c825-4f2c-a6ce-70e06f68f187-sys\") pod \"tuned-dr9c2\" (UID: \"87f77489-c825-4f2c-a6ce-70e06f68f187\") " pod="openshift-cluster-node-tuning-operator/tuned-dr9c2"
Oct 30 22:06:18.841251 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.839462    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/015f218d-bf1f-4e7e-bb5d-b805af003246-ovnkube-config\") pod \"ovnkube-node-jcf8l\" (UID: \"015f218d-bf1f-4e7e-bb5d-b805af003246\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l"
Oct 30 22:06:18.841251 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.840712    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/87f77489-c825-4f2c-a6ce-70e06f68f187-etc-modprobe-d\") pod \"tuned-dr9c2\" (UID: \"87f77489-c825-4f2c-a6ce-70e06f68f187\") " pod="openshift-cluster-node-tuning-operator/tuned-dr9c2"
Oct 30 22:06:18.841251 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.840696    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0e128ead-13e5-41f3-9ae7-6f944b5f18ee-sys-fs\") pod \"aws-ebs-csi-driver-node-bbkdz\" (UID: \"0e128ead-13e5-41f3-9ae7-6f944b5f18ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bbkdz"
Oct 30 22:06:18.841251 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.840730    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/87f77489-c825-4f2c-a6ce-70e06f68f187-run\") pod \"tuned-dr9c2\" (UID: \"87f77489-c825-4f2c-a6ce-70e06f68f187\") " pod="openshift-cluster-node-tuning-operator/tuned-dr9c2"
Oct 30 22:06:18.841251 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.840762    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/15e3a459-855f-4181-825c-345e9b2e62e1-cni-binary-copy\") pod \"multus-additional-cni-plugins-22c27\" (UID: \"15e3a459-855f-4181-825c-345e9b2e62e1\") " pod="openshift-multus/multus-additional-cni-plugins-22c27"
Oct 30 22:06:18.841251 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.840775    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/87f77489-c825-4f2c-a6ce-70e06f68f187-run\") pod \"tuned-dr9c2\" (UID: \"87f77489-c825-4f2c-a6ce-70e06f68f187\") " pod="openshift-cluster-node-tuning-operator/tuned-dr9c2"
Oct 30 22:06:18.841251 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.840823    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7b50e755-6b0b-4d45-b7d3-c822127e6cb1-system-cni-dir\") pod \"multus-4s4vc\" (UID: \"7b50e755-6b0b-4d45-b7d3-c822127e6cb1\") " pod="openshift-multus/multus-4s4vc"
Oct 30 22:06:18.841251 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.840848    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e9c20fdf-bea5-4d6c-b252-b29b08bfdf63-serviceca\") pod \"node-ca-b74lk\" (UID: \"e9c20fdf-bea5-4d6c-b252-b29b08bfdf63\") " pod="openshift-image-registry/node-ca-b74lk"
Oct 30 22:06:18.841251 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.840890    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57q7f\" (UniqueName: \"kubernetes.io/projected/015f218d-bf1f-4e7e-bb5d-b805af003246-kube-api-access-57q7f\") pod \"ovnkube-node-jcf8l\" (UID: \"015f218d-bf1f-4e7e-bb5d-b805af003246\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l"
Oct 30 22:06:18.841251 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.840915    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0e128ead-13e5-41f3-9ae7-6f944b5f18ee-device-dir\") pod \"aws-ebs-csi-driver-node-bbkdz\" (UID: \"0e128ead-13e5-41f3-9ae7-6f944b5f18ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bbkdz"
Oct 30 22:06:18.841251 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.840962    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/87f77489-c825-4f2c-a6ce-70e06f68f187-etc-kubernetes\") pod \"tuned-dr9c2\" (UID: \"87f77489-c825-4f2c-a6ce-70e06f68f187\") " pod="openshift-cluster-node-tuning-operator/tuned-dr9c2"
Oct 30 22:06:18.842016 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.840984    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/87f77489-c825-4f2c-a6ce-70e06f68f187-var-lib-kubelet\") pod \"tuned-dr9c2\" (UID: \"87f77489-c825-4f2c-a6ce-70e06f68f187\") " pod="openshift-cluster-node-tuning-operator/tuned-dr9c2"
Oct 30 22:06:18.842016 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.841016    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b8503b30-31d6-4454-a3fd-8bea4a527b04-original-pull-secret\") pod \"global-pull-secret-syncer-rjtc8\" (UID: \"b8503b30-31d6-4454-a3fd-8bea4a527b04\") " pod="kube-system/global-pull-secret-syncer-rjtc8"
Oct 30 22:06:18.842016 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.841056    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frxsh\" (UniqueName: \"kubernetes.io/projected/fe4c45fd-690b-4111-9d15-93dc183b3e46-kube-api-access-frxsh\") pod \"network-metrics-daemon-6hhw9\" (UID: \"fe4c45fd-690b-4111-9d15-93dc183b3e46\") " pod="openshift-multus/network-metrics-daemon-6hhw9"
Oct 30 22:06:18.842016 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.841081    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e9c20fdf-bea5-4d6c-b252-b29b08bfdf63-host\") pod \"node-ca-b74lk\" (UID: \"e9c20fdf-bea5-4d6c-b252-b29b08bfdf63\") " pod="openshift-image-registry/node-ca-b74lk"
Oct 30 22:06:18.842016 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.841127    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/015f218d-bf1f-4e7e-bb5d-b805af003246-run-systemd\") pod \"ovnkube-node-jcf8l\" (UID: \"015f218d-bf1f-4e7e-bb5d-b805af003246\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l"
Oct 30 22:06:18.842016 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.841165    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/015f218d-bf1f-4e7e-bb5d-b805af003246-host-cni-bin\") pod \"ovnkube-node-jcf8l\" (UID: \"015f218d-bf1f-4e7e-bb5d-b805af003246\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l"
Oct 30 22:06:18.842016 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.841187    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/87f77489-c825-4f2c-a6ce-70e06f68f187-etc-sysctl-d\") pod \"tuned-dr9c2\" (UID: \"87f77489-c825-4f2c-a6ce-70e06f68f187\") " pod="openshift-cluster-node-tuning-operator/tuned-dr9c2"
Oct 30 22:06:18.842016 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.841241    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7b50e755-6b0b-4d45-b7d3-c822127e6cb1-multus-daemon-config\") pod \"multus-4s4vc\" (UID: \"7b50e755-6b0b-4d45-b7d3-c822127e6cb1\") " pod="openshift-multus/multus-4s4vc"
Oct 30 22:06:18.842016 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.841264    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cdcp\" (UniqueName: \"kubernetes.io/projected/e9c20fdf-bea5-4d6c-b252-b29b08bfdf63-kube-api-access-5cdcp\") pod \"node-ca-b74lk\" (UID: \"e9c20fdf-bea5-4d6c-b252-b29b08bfdf63\") " pod="openshift-image-registry/node-ca-b74lk"
Oct 30 22:06:18.842016 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.841285    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/015f218d-bf1f-4e7e-bb5d-b805af003246-log-socket\") pod \"ovnkube-node-jcf8l\" (UID: \"015f218d-bf1f-4e7e-bb5d-b805af003246\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l"
Oct 30 22:06:18.842016 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.841329    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0e128ead-13e5-41f3-9ae7-6f944b5f18ee-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bbkdz\" (UID: \"0e128ead-13e5-41f3-9ae7-6f944b5f18ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bbkdz"
Oct 30 22:06:18.842016 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.841387    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7b50e755-6b0b-4d45-b7d3-c822127e6cb1-system-cni-dir\") pod \"multus-4s4vc\" (UID: \"7b50e755-6b0b-4d45-b7d3-c822127e6cb1\") " pod="openshift-multus/multus-4s4vc"
Oct 30 22:06:18.842016 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.841464    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/015f218d-bf1f-4e7e-bb5d-b805af003246-run-systemd\") pod \"ovnkube-node-jcf8l\" (UID: \"015f218d-bf1f-4e7e-bb5d-b805af003246\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l"
Oct 30 22:06:18.842016 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.841514    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e9c20fdf-bea5-4d6c-b252-b29b08bfdf63-host\") pod \"node-ca-b74lk\" (UID: \"e9c20fdf-bea5-4d6c-b252-b29b08bfdf63\") " pod="openshift-image-registry/node-ca-b74lk"
Oct 30 22:06:18.842016 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.841562    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/87f77489-c825-4f2c-a6ce-70e06f68f187-etc-kubernetes\") pod \"tuned-dr9c2\" (UID: \"87f77489-c825-4f2c-a6ce-70e06f68f187\") " pod="openshift-cluster-node-tuning-operator/tuned-dr9c2"
Oct 30 22:06:18.842016 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.841563    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/87f77489-c825-4f2c-a6ce-70e06f68f187-etc-sysctl-d\") pod \"tuned-dr9c2\" (UID: \"87f77489-c825-4f2c-a6ce-70e06f68f187\") " pod="openshift-cluster-node-tuning-operator/tuned-dr9c2"
Oct 30 22:06:18.842016 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.841643    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0e128ead-13e5-41f3-9ae7-6f944b5f18ee-device-dir\") pod \"aws-ebs-csi-driver-node-bbkdz\" (UID: \"0e128ead-13e5-41f3-9ae7-6f944b5f18ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bbkdz"
Oct 30 22:06:18.842016 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.841703    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/87f77489-c825-4f2c-a6ce-70e06f68f187-lib-modules\") pod \"tuned-dr9c2\" (UID: \"87f77489-c825-4f2c-a6ce-70e06f68f187\") " pod="openshift-cluster-node-tuning-operator/tuned-dr9c2"
Oct 30 22:06:18.842744 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.841745    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/015f218d-bf1f-4e7e-bb5d-b805af003246-node-log\") pod \"ovnkube-node-jcf8l\" (UID: \"015f218d-bf1f-4e7e-bb5d-b805af003246\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l"
Oct 30 22:06:18.842744 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.841769    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7b50e755-6b0b-4d45-b7d3-c822127e6cb1-etc-kubernetes\") pod \"multus-4s4vc\" (UID: \"7b50e755-6b0b-4d45-b7d3-c822127e6cb1\") " pod="openshift-multus/multus-4s4vc"
Oct 30 22:06:18.842744 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.841793    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/015f218d-bf1f-4e7e-bb5d-b805af003246-ovnkube-script-lib\") pod \"ovnkube-node-jcf8l\" (UID: \"015f218d-bf1f-4e7e-bb5d-b805af003246\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l"
Oct 30 22:06:18.842744 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.841815    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/87f77489-c825-4f2c-a6ce-70e06f68f187-tmp\") pod \"tuned-dr9c2\" (UID: \"87f77489-c825-4f2c-a6ce-70e06f68f187\") " pod="openshift-cluster-node-tuning-operator/tuned-dr9c2"
Oct 30 22:06:18.842744 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.841844    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/15e3a459-855f-4181-825c-345e9b2e62e1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-22c27\" (UID: \"15e3a459-855f-4181-825c-345e9b2e62e1\") " pod="openshift-multus/multus-additional-cni-plugins-22c27"
Oct 30 22:06:18.842744 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.841891    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7b50e755-6b0b-4d45-b7d3-c822127e6cb1-etc-kubernetes\") pod \"multus-4s4vc\" (UID: \"7b50e755-6b0b-4d45-b7d3-c822127e6cb1\") " pod="openshift-multus/multus-4s4vc"
Oct 30 22:06:18.842744 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.841905    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7b50e755-6b0b-4d45-b7d3-c822127e6cb1-multus-cni-dir\") pod \"multus-4s4vc\" (UID: \"7b50e755-6b0b-4d45-b7d3-c822127e6cb1\") " pod="openshift-multus/multus-4s4vc"
Oct 30 22:06:18.842744 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.841950    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/015f218d-bf1f-4e7e-bb5d-b805af003246-node-log\") pod \"ovnkube-node-jcf8l\" (UID: \"015f218d-bf1f-4e7e-bb5d-b805af003246\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l"
Oct 30 22:06:18.842744 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.841967    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da9b75bf-8af2-450f-9a98-c265eea48282-tmp-dir\") pod \"node-resolver-4knxq\" (UID: \"da9b75bf-8af2-450f-9a98-c265eea48282\") " pod="openshift-dns/node-resolver-4knxq"
Oct 30 22:06:18.842744 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.841990    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/87f77489-c825-4f2c-a6ce-70e06f68f187-lib-modules\") pod \"tuned-dr9c2\" (UID: \"87f77489-c825-4f2c-a6ce-70e06f68f187\") " pod="openshift-cluster-node-tuning-operator/tuned-dr9c2"
Oct 30 22:06:18.843176 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.843052    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e9c20fdf-bea5-4d6c-b252-b29b08bfdf63-serviceca\") pod \"node-ca-b74lk\" (UID: \"e9c20fdf-bea5-4d6c-b252-b29b08bfdf63\") " pod="openshift-image-registry/node-ca-b74lk"
Oct 30 22:06:18.843176 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.843112    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/015f218d-bf1f-4e7e-bb5d-b805af003246-ovnkube-script-lib\") pod \"ovnkube-node-jcf8l\" (UID: \"015f218d-bf1f-4e7e-bb5d-b805af003246\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l"
Oct 30 22:06:18.843176 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.841993    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8286n\" (UniqueName: \"kubernetes.io/projected/dbf6b14c-5576-4ae3-bbb3-b3201aa7cb7c-kube-api-access-8286n\") pod \"network-check-target-vs5pt\" (UID: \"dbf6b14c-5576-4ae3-bbb3-b3201aa7cb7c\") " pod="openshift-network-diagnostics/network-check-target-vs5pt"
Oct 30 22:06:18.843319 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.843237    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/015f218d-bf1f-4e7e-bb5d-b805af003246-host-run-netns\") pod \"ovnkube-node-jcf8l\" (UID: \"015f218d-bf1f-4e7e-bb5d-b805af003246\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l"
Oct 30 22:06:18.843379 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.843337    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/015f218d-bf1f-4e7e-bb5d-b805af003246-run-ovn\") pod \"ovnkube-node-jcf8l\" (UID: \"015f218d-bf1f-4e7e-bb5d-b805af003246\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l"
Oct 30 22:06:18.843379 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.843366    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/87f77489-c825-4f2c-a6ce-70e06f68f187-etc-sysconfig\") pod \"tuned-dr9c2\" (UID: \"87f77489-c825-4f2c-a6ce-70e06f68f187\") " pod="openshift-cluster-node-tuning-operator/tuned-dr9c2"
Oct 30 22:06:18.843465 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:18.843410    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 22:06:18.843465 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.843412    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/15e3a459-855f-4181-825c-345e9b2e62e1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-22c27\" (UID: \"15e3a459-855f-4181-825c-345e9b2e62e1\") " pod="openshift-multus/multus-additional-cni-plugins-22c27"
Oct 30 22:06:18.843465 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:18.843459    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8503b30-31d6-4454-a3fd-8bea4a527b04-original-pull-secret podName:b8503b30-31d6-4454-a3fd-8bea4a527b04 nodeName:}" failed. No retries permitted until 2026-10-30 22:06:19.343442606 +0000 UTC m=+3.096796828 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b8503b30-31d6-4454-a3fd-8bea4a527b04-original-pull-secret") pod "global-pull-secret-syncer-rjtc8" (UID: "b8503b30-31d6-4454-a3fd-8bea4a527b04") : object "kube-system"/"original-pull-secret" not registered
Oct 30 22:06:18.843622 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.843476    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ncxp\" (UniqueName: \"kubernetes.io/projected/15e3a459-855f-4181-825c-345e9b2e62e1-kube-api-access-4ncxp\") pod \"multus-additional-cni-plugins-22c27\" (UID: \"15e3a459-855f-4181-825c-345e9b2e62e1\") " pod="openshift-multus/multus-additional-cni-plugins-22c27"
Oct 30 22:06:18.843622 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.843505    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkshh\" (UniqueName: \"kubernetes.io/projected/da9b75bf-8af2-450f-9a98-c265eea48282-kube-api-access-mkshh\") pod \"node-resolver-4knxq\" (UID: \"da9b75bf-8af2-450f-9a98-c265eea48282\") " pod="openshift-dns/node-resolver-4knxq"
Oct 30 22:06:18.843622 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.843527    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2e4eaf5f-8a83-4681-b040-65676affdf7e-host-slash\") pod \"iptables-alerter-vw5fg\" (UID: \"2e4eaf5f-8a83-4681-b040-65676affdf7e\") " pod="openshift-network-operator/iptables-alerter-vw5fg"
Oct 30 22:06:18.843622 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.843544    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0e128ead-13e5-41f3-9ae7-6f944b5f18ee-etc-selinux\") pod \"aws-ebs-csi-driver-node-bbkdz\" (UID: \"0e128ead-13e5-41f3-9ae7-6f944b5f18ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bbkdz"
Oct 30 22:06:18.843622 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.843564    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7b50e755-6b0b-4d45-b7d3-c822127e6cb1-host-var-lib-cni-bin\") pod \"multus-4s4vc\" (UID: \"7b50e755-6b0b-4d45-b7d3-c822127e6cb1\") " pod="openshift-multus/multus-4s4vc"
Oct 30 22:06:18.843622 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.843585    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2e4eaf5f-8a83-4681-b040-65676affdf7e-iptables-alerter-script\") pod \"iptables-alerter-vw5fg\" (UID: \"2e4eaf5f-8a83-4681-b040-65676affdf7e\") " pod="openshift-network-operator/iptables-alerter-vw5fg"
Oct 30 22:06:18.843622 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.843604    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/87f77489-c825-4f2c-a6ce-70e06f68f187-etc-tuned\") pod \"tuned-dr9c2\" (UID: \"87f77489-c825-4f2c-a6ce-70e06f68f187\") " pod="openshift-cluster-node-tuning-operator/tuned-dr9c2"
Oct 30 22:06:18.843622 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.843620    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/15e3a459-855f-4181-825c-345e9b2e62e1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-22c27\" (UID: \"15e3a459-855f-4181-825c-345e9b2e62e1\") " pod="openshift-multus/multus-additional-cni-plugins-22c27"
Oct 30 22:06:18.844012 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.843639    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b8503b30-31d6-4454-a3fd-8bea4a527b04-dbus\") pod \"global-pull-secret-syncer-rjtc8\" (UID: \"b8503b30-31d6-4454-a3fd-8bea4a527b04\") " pod="kube-system/global-pull-secret-syncer-rjtc8"
Oct 30 22:06:18.844012 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.843657    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7b50e755-6b0b-4d45-b7d3-c822127e6cb1-cnibin\") pod \"multus-4s4vc\" (UID: \"7b50e755-6b0b-4d45-b7d3-c822127e6cb1\") " pod="openshift-multus/multus-4s4vc"
Oct 30 22:06:18.844012 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.843680    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7b50e755-6b0b-4d45-b7d3-c822127e6cb1-host-var-lib-cni-multus\") pod \"multus-4s4vc\" (UID: \"7b50e755-6b0b-4d45-b7d3-c822127e6cb1\") " pod="openshift-multus/multus-4s4vc"
Oct 30 22:06:18.844012 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.843701    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7b50e755-6b0b-4d45-b7d3-c822127e6cb1-multus-conf-dir\") pod \"multus-4s4vc\" (UID: \"7b50e755-6b0b-4d45-b7d3-c822127e6cb1\") " pod="openshift-multus/multus-4s4vc"
Oct 30 22:06:18.844012 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.843722    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/da9b75bf-8af2-450f-9a98-c265eea48282-hosts-file\") pod \"node-resolver-4knxq\" (UID: \"da9b75bf-8af2-450f-9a98-c265eea48282\") " pod="openshift-dns/node-resolver-4knxq"
Oct 30 22:06:18.844012 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.843742    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/15e3a459-855f-4181-825c-345e9b2e62e1-cni-binary-copy\") pod \"multus-additional-cni-plugins-22c27\" (UID: \"15e3a459-855f-4181-825c-345e9b2e62e1\") " pod="openshift-multus/multus-additional-cni-plugins-22c27"
Oct 30 22:06:18.844012 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.843748    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/015f218d-bf1f-4e7e-bb5d-b805af003246-etc-openvswitch\") pod \"ovnkube-node-jcf8l\" (UID: \"015f218d-bf1f-4e7e-bb5d-b805af003246\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l"
Oct 30 22:06:18.844012 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.843785    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/015f218d-bf1f-4e7e-bb5d-b805af003246-etc-openvswitch\") pod \"ovnkube-node-jcf8l\" (UID: \"015f218d-bf1f-4e7e-bb5d-b805af003246\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l"
Oct 30 22:06:18.844012 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.843795    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mn4fl\" (UniqueName: \"kubernetes.io/projected/0e128ead-13e5-41f3-9ae7-6f944b5f18ee-kube-api-access-mn4fl\") pod \"aws-ebs-csi-driver-node-bbkdz\" (UID: \"0e128ead-13e5-41f3-9ae7-6f944b5f18ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bbkdz"
Oct 30 22:06:18.844012 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.843921    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/15e3a459-855f-4181-825c-345e9b2e62e1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-22c27\" (UID: \"15e3a459-855f-4181-825c-345e9b2e62e1\") " pod="openshift-multus/multus-additional-cni-plugins-22c27"
Oct 30 22:06:18.844564 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.844159    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2e4eaf5f-8a83-4681-b040-65676affdf7e-host-slash\") pod \"iptables-alerter-vw5fg\" (UID: \"2e4eaf5f-8a83-4681-b040-65676affdf7e\") " pod="openshift-network-operator/iptables-alerter-vw5fg"
Oct 30 22:06:18.844564 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.844291    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7b50e755-6b0b-4d45-b7d3-c822127e6cb1-host-var-lib-cni-bin\") pod \"multus-4s4vc\" (UID: \"7b50e755-6b0b-4d45-b7d3-c822127e6cb1\") " pod="openshift-multus/multus-4s4vc"
Oct 30 22:06:18.844564 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.844433    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/015f218d-bf1f-4e7e-bb5d-b805af003246-run-ovn\") pod \"ovnkube-node-jcf8l\" (UID: \"015f218d-bf1f-4e7e-bb5d-b805af003246\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l"
Oct 30 22:06:18.844564 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.844506    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/87f77489-c825-4f2c-a6ce-70e06f68f187-etc-sysconfig\") pod \"tuned-dr9c2\" (UID: \"87f77489-c825-4f2c-a6ce-70e06f68f187\") " pod="openshift-cluster-node-tuning-operator/tuned-dr9c2"
Oct 30 22:06:18.844771 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.844748    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/015f218d-bf1f-4e7e-bb5d-b805af003246-host-run-netns\") pod \"ovnkube-node-jcf8l\" (UID: \"015f218d-bf1f-4e7e-bb5d-b805af003246\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l"
Oct 30 22:06:18.844975 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.844955    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a3d96078-3ec8-40aa-855a-d1ba4acf2e74-agent-certs\") pod \"konnectivity-agent-mm655\" (UID: \"a3d96078-3ec8-40aa-855a-d1ba4acf2e74\") " pod="kube-system/konnectivity-agent-mm655"
Oct 30 22:06:18.845206 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.845093    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0e128ead-13e5-41f3-9ae7-6f944b5f18ee-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bbkdz\" (UID: \"0e128ead-13e5-41f3-9ae7-6f944b5f18ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bbkdz"
Oct 30 22:06:18.845206 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.841879    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7b50e755-6b0b-4d45-b7d3-c822127e6cb1-cni-binary-copy\") pod \"multus-4s4vc\" (UID: \"7b50e755-6b0b-4d45-b7d3-c822127e6cb1\") " pod="openshift-multus/multus-4s4vc"
Oct 30 22:06:18.845206 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.845123    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/015f218d-bf1f-4e7e-bb5d-b805af003246-host-cni-bin\") pod \"ovnkube-node-jcf8l\" (UID: \"015f218d-bf1f-4e7e-bb5d-b805af003246\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l"
Oct 30 22:06:18.845595 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.845247    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/87f77489-c825-4f2c-a6ce-70e06f68f187-var-lib-kubelet\") pod \"tuned-dr9c2\" (UID: \"87f77489-c825-4f2c-a6ce-70e06f68f187\") " pod="openshift-cluster-node-tuning-operator/tuned-dr9c2"
Oct 30 22:06:18.845595 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.845305    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7b50e755-6b0b-4d45-b7d3-c822127e6cb1-multus-conf-dir\") pod \"multus-4s4vc\" (UID: \"7b50e755-6b0b-4d45-b7d3-c822127e6cb1\") " pod="openshift-multus/multus-4s4vc"
Oct 30 22:06:18.845595 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.842007    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/015f218d-bf1f-4e7e-bb5d-b805af003246-log-socket\") pod \"ovnkube-node-jcf8l\" (UID: \"015f218d-bf1f-4e7e-bb5d-b805af003246\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l"
Oct 30 22:06:18.845595 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.845425    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7b50e755-6b0b-4d45-b7d3-c822127e6cb1-cnibin\") pod \"multus-4s4vc\" (UID: \"7b50e755-6b0b-4d45-b7d3-c822127e6cb1\") " pod="openshift-multus/multus-4s4vc"
Oct 30 22:06:18.845595 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.845435    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/da9b75bf-8af2-450f-9a98-c265eea48282-hosts-file\") pod \"node-resolver-4knxq\" (UID: \"da9b75bf-8af2-450f-9a98-c265eea48282\") " pod="openshift-dns/node-resolver-4knxq"
Oct 30 22:06:18.845595 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.845529    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b8503b30-31d6-4454-a3fd-8bea4a527b04-dbus\") pod \"global-pull-secret-syncer-rjtc8\" (UID: \"b8503b30-31d6-4454-a3fd-8bea4a527b04\") " pod="kube-system/global-pull-secret-syncer-rjtc8"
Oct 30 22:06:18.845595 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.845569    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7b50e755-6b0b-4d45-b7d3-c822127e6cb1-multus-daemon-config\") pod \"multus-4s4vc\" (UID: \"7b50e755-6b0b-4d45-b7d3-c822127e6cb1\") " pod="openshift-multus/multus-4s4vc"
Oct 30 22:06:18.845906 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.845648    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2e4eaf5f-8a83-4681-b040-65676affdf7e-iptables-alerter-script\") pod \"iptables-alerter-vw5fg\" (UID: \"2e4eaf5f-8a83-4681-b040-65676affdf7e\") " pod="openshift-network-operator/iptables-alerter-vw5fg"
Oct 30 22:06:18.845906 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.845679    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/15e3a459-855f-4181-825c-345e9b2e62e1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-22c27\" (UID: \"15e3a459-855f-4181-825c-345e9b2e62e1\") " pod="openshift-multus/multus-additional-cni-plugins-22c27"
Oct 30 22:06:18.845906 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.845767    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7b50e755-6b0b-4d45-b7d3-c822127e6cb1-multus-cni-dir\") pod \"multus-4s4vc\" (UID: \"7b50e755-6b0b-4d45-b7d3-c822127e6cb1\") " pod="openshift-multus/multus-4s4vc"
Oct 30 22:06:18.846076 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.846058    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da9b75bf-8af2-450f-9a98-c265eea48282-tmp-dir\") pod \"node-resolver-4knxq\" (UID: \"da9b75bf-8af2-450f-9a98-c265eea48282\") " pod="openshift-dns/node-resolver-4knxq"
Oct 30 22:06:18.846200 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.846176    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/15e3a459-855f-4181-825c-345e9b2e62e1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-22c27\" (UID: \"15e3a459-855f-4181-825c-345e9b2e62e1\") " pod="openshift-multus/multus-additional-cni-plugins-22c27"
Oct 30 22:06:18.846273 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.846259    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0e128ead-13e5-41f3-9ae7-6f944b5f18ee-etc-selinux\") pod \"aws-ebs-csi-driver-node-bbkdz\" (UID: \"0e128ead-13e5-41f3-9ae7-6f944b5f18ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bbkdz"
Oct 30 22:06:18.846393 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.846359    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/015f218d-bf1f-4e7e-bb5d-b805af003246-ovn-node-metrics-cert\") pod \"ovnkube-node-jcf8l\" (UID: \"015f218d-bf1f-4e7e-bb5d-b805af003246\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l"
Oct 30 22:06:18.847021 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.847000    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/87f77489-c825-4f2c-a6ce-70e06f68f187-etc-tuned\") pod \"tuned-dr9c2\" (UID: \"87f77489-c825-4f2c-a6ce-70e06f68f187\") " pod="openshift-cluster-node-tuning-operator/tuned-dr9c2"
Oct 30 22:06:18.848293 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.848240    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7b50e755-6b0b-4d45-b7d3-c822127e6cb1-host-var-lib-cni-multus\") pod \"multus-4s4vc\" (UID: \"7b50e755-6b0b-4d45-b7d3-c822127e6cb1\") " pod="openshift-multus/multus-4s4vc"
Oct 30 22:06:18.848703 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.848679    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/87f77489-c825-4f2c-a6ce-70e06f68f187-tmp\") pod \"tuned-dr9c2\" (UID: \"87f77489-c825-4f2c-a6ce-70e06f68f187\") " pod="openshift-cluster-node-tuning-operator/tuned-dr9c2"
Oct 30 22:06:18.861705 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.861678    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5d9bb\" (UniqueName: \"kubernetes.io/projected/7b50e755-6b0b-4d45-b7d3-c822127e6cb1-kube-api-access-5d9bb\") pod \"multus-4s4vc\" (UID: \"7b50e755-6b0b-4d45-b7d3-c822127e6cb1\") " pod="openshift-multus/multus-4s4vc"
Oct 30 22:06:18.862752 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:18.862733    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 22:06:18.862888 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:18.862875    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 22:06:18.863004 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:18.862988    2579 projected.go:196] Error preparing data for projected volume kube-api-access-8286n for pod openshift-network-diagnostics/network-check-target-vs5pt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 22:06:18.863004 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.863003    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfxls\" (UniqueName: \"kubernetes.io/projected/2e4eaf5f-8a83-4681-b040-65676affdf7e-kube-api-access-tfxls\") pod \"iptables-alerter-vw5fg\" (UID: \"2e4eaf5f-8a83-4681-b040-65676affdf7e\") " pod="openshift-network-operator/iptables-alerter-vw5fg"
Oct 30 22:06:18.863173 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:18.863073    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dbf6b14c-5576-4ae3-bbb3-b3201aa7cb7c-kube-api-access-8286n podName:dbf6b14c-5576-4ae3-bbb3-b3201aa7cb7c nodeName:}" failed. No retries permitted until 2026-10-30 22:06:19.363055745 +0000 UTC m=+3.116409987 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-8286n" (UniqueName: "kubernetes.io/projected/dbf6b14c-5576-4ae3-bbb3-b3201aa7cb7c-kube-api-access-8286n") pod "network-check-target-vs5pt" (UID: "dbf6b14c-5576-4ae3-bbb3-b3201aa7cb7c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 22:06:18.865075 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.865034    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hc5rj\" (UniqueName: \"kubernetes.io/projected/87f77489-c825-4f2c-a6ce-70e06f68f187-kube-api-access-hc5rj\") pod \"tuned-dr9c2\" (UID: \"87f77489-c825-4f2c-a6ce-70e06f68f187\") " pod="openshift-cluster-node-tuning-operator/tuned-dr9c2"
Oct 30 22:06:18.865156 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.865092    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cdcp\" (UniqueName: \"kubernetes.io/projected/e9c20fdf-bea5-4d6c-b252-b29b08bfdf63-kube-api-access-5cdcp\") pod \"node-ca-b74lk\" (UID: \"e9c20fdf-bea5-4d6c-b252-b29b08bfdf63\") " pod="openshift-image-registry/node-ca-b74lk"
Oct 30 22:06:18.865705 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.865655    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frxsh\" (UniqueName: \"kubernetes.io/projected/fe4c45fd-690b-4111-9d15-93dc183b3e46-kube-api-access-frxsh\") pod \"network-metrics-daemon-6hhw9\" (UID: \"fe4c45fd-690b-4111-9d15-93dc183b3e46\") " pod="openshift-multus/network-metrics-daemon-6hhw9"
Oct 30 22:06:18.866486 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.866464    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkshh\" (UniqueName: \"kubernetes.io/projected/da9b75bf-8af2-450f-9a98-c265eea48282-kube-api-access-mkshh\") pod \"node-resolver-4knxq\" (UID: \"da9b75bf-8af2-450f-9a98-c265eea48282\") " pod="openshift-dns/node-resolver-4knxq"
Oct 30 22:06:18.867142 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.867123    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ncxp\" (UniqueName: \"kubernetes.io/projected/15e3a459-855f-4181-825c-345e9b2e62e1-kube-api-access-4ncxp\") pod \"multus-additional-cni-plugins-22c27\" (UID: \"15e3a459-855f-4181-825c-345e9b2e62e1\") " pod="openshift-multus/multus-additional-cni-plugins-22c27"
Oct 30 22:06:18.867632 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.867589    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mn4fl\" (UniqueName: \"kubernetes.io/projected/0e128ead-13e5-41f3-9ae7-6f944b5f18ee-kube-api-access-mn4fl\") pod \"aws-ebs-csi-driver-node-bbkdz\" (UID: \"0e128ead-13e5-41f3-9ae7-6f944b5f18ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bbkdz"
Oct 30 22:06:18.868439 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.868416    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57q7f\" (UniqueName: \"kubernetes.io/projected/015f218d-bf1f-4e7e-bb5d-b805af003246-kube-api-access-57q7f\") pod \"ovnkube-node-jcf8l\" (UID: \"015f218d-bf1f-4e7e-bb5d-b805af003246\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l"
Oct 30 22:06:18.952264 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.952243    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bbkdz"
Oct 30 22:06:18.955368 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.955342    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dr9c2"
Oct 30 22:06:18.964321 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:18.964296    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e128ead_13e5_41f3_9ae7_6f944b5f18ee.slice/crio-f47cd69ac5da515bb68e1c56a99c33e22b32498bf9de466ca92af4f71088dae2 WatchSource:0}: Error finding container f47cd69ac5da515bb68e1c56a99c33e22b32498bf9de466ca92af4f71088dae2: Status 404 returned error can't find the container with id f47cd69ac5da515bb68e1c56a99c33e22b32498bf9de466ca92af4f71088dae2
Oct 30 22:06:18.967890 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.967872    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mm655"
Oct 30 22:06:18.968014 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:18.967990    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87f77489_c825_4f2c_a6ce_70e06f68f187.slice/crio-8fa2f930c7a689eefaa2d8987458a7101eb0907572b0cbafb8be10af6fb91d36 WatchSource:0}: Error finding container 8fa2f930c7a689eefaa2d8987458a7101eb0907572b0cbafb8be10af6fb91d36: Status 404 returned error can't find the container with id 8fa2f930c7a689eefaa2d8987458a7101eb0907572b0cbafb8be10af6fb91d36
Oct 30 22:06:18.978854 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:18.978829    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3d96078_3ec8_40aa_855a_d1ba4acf2e74.slice/crio-328a6e8e630e43c5536d50c0c617888a400f575042f78833b718b6ab8188c58d WatchSource:0}: Error finding container 328a6e8e630e43c5536d50c0c617888a400f575042f78833b718b6ab8188c58d: Status 404 returned error can't find the container with id 328a6e8e630e43c5536d50c0c617888a400f575042f78833b718b6ab8188c58d
Oct 30 22:06:18.986503 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.986342    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-22c27"
Oct 30 22:06:18.988688 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.988670    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4s4vc"
Oct 30 22:06:18.993382 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.993337    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-b74lk"
Oct 30 22:06:18.997492 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:18.997466    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15e3a459_855f_4181_825c_345e9b2e62e1.slice/crio-e0346fe028ed362ec3b464577562de2ae63214a8eaf5f062879fd52f0b3ca089 WatchSource:0}: Error finding container e0346fe028ed362ec3b464577562de2ae63214a8eaf5f062879fd52f0b3ca089: Status 404 returned error can't find the container with id e0346fe028ed362ec3b464577562de2ae63214a8eaf5f062879fd52f0b3ca089
Oct 30 22:06:18.998243 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:18.998224    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4knxq"
Oct 30 22:06:18.999424 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:18.999393    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b50e755_6b0b_4d45_b7d3_c822127e6cb1.slice/crio-45efc55eedbfd6c64303396ae3d213a3b6cf4e9f53fc0a0bab6ab516227fe3c5 WatchSource:0}: Error finding container 45efc55eedbfd6c64303396ae3d213a3b6cf4e9f53fc0a0bab6ab516227fe3c5: Status 404 returned error can't find the container with id 45efc55eedbfd6c64303396ae3d213a3b6cf4e9f53fc0a0bab6ab516227fe3c5
Oct 30 22:06:19.005934 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:19.005914    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vw5fg"
Oct 30 22:06:19.013069 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:19.011102    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9c20fdf_bea5_4d6c_b252_b29b08bfdf63.slice/crio-f93b6fcd91ff7bdca9eb5ca4eb05183ad2ac1e9b0da47e40d0bc8be3748fd2ec WatchSource:0}: Error finding container f93b6fcd91ff7bdca9eb5ca4eb05183ad2ac1e9b0da47e40d0bc8be3748fd2ec: Status 404 returned error can't find the container with id f93b6fcd91ff7bdca9eb5ca4eb05183ad2ac1e9b0da47e40d0bc8be3748fd2ec
Oct 30 22:06:19.016920 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:19.015980    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda9b75bf_8af2_450f_9a98_c265eea48282.slice/crio-faaad9c80829f8bc4d3a4972a649291ae22f1ad8792b1393f1aef56194fb3f56 WatchSource:0}: Error finding container faaad9c80829f8bc4d3a4972a649291ae22f1ad8792b1393f1aef56194fb3f56: Status 404 returned error can't find the container with id faaad9c80829f8bc4d3a4972a649291ae22f1ad8792b1393f1aef56194fb3f56
Oct 30 22:06:19.019165 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:19.019141    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e4eaf5f_8a83_4681_b040_65676affdf7e.slice/crio-285e3b8fbd1b072ec18318062c4f11065437f96d1131aaf0ec321e4aa459a467 WatchSource:0}: Error finding container 285e3b8fbd1b072ec18318062c4f11065437f96d1131aaf0ec321e4aa459a467: Status 404 returned error can't find the container with id 285e3b8fbd1b072ec18318062c4f11065437f96d1131aaf0ec321e4aa459a467
Oct 30 22:06:19.026876 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:19.026848    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l"
Oct 30 22:06:19.040450 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:19.040428    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod015f218d_bf1f_4e7e_bb5d_b805af003246.slice/crio-eaf4f0878c59e7dec85bba1fc9a00bd5871a054e5b82711075cba359d8242920 WatchSource:0}: Error finding container eaf4f0878c59e7dec85bba1fc9a00bd5871a054e5b82711075cba359d8242920: Status 404 returned error can't find the container with id eaf4f0878c59e7dec85bba1fc9a00bd5871a054e5b82711075cba359d8242920
Oct 30 22:06:19.348882 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:19.348701    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe4c45fd-690b-4111-9d15-93dc183b3e46-metrics-certs\") pod \"network-metrics-daemon-6hhw9\" (UID: \"fe4c45fd-690b-4111-9d15-93dc183b3e46\") " pod="openshift-multus/network-metrics-daemon-6hhw9"
Oct 30 22:06:19.348995 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:19.348929    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b8503b30-31d6-4454-a3fd-8bea4a527b04-original-pull-secret\") pod \"global-pull-secret-syncer-rjtc8\" (UID: \"b8503b30-31d6-4454-a3fd-8bea4a527b04\") " pod="kube-system/global-pull-secret-syncer-rjtc8"
Oct 30 22:06:19.349077 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:19.349025    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 22:06:19.349077 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:19.349071    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 22:06:19.349163 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:19.349112    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe4c45fd-690b-4111-9d15-93dc183b3e46-metrics-certs podName:fe4c45fd-690b-4111-9d15-93dc183b3e46 nodeName:}" failed. No retries permitted until 2026-10-30 22:06:20.349091688 +0000 UTC m=+4.102445912 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fe4c45fd-690b-4111-9d15-93dc183b3e46-metrics-certs") pod "network-metrics-daemon-6hhw9" (UID: "fe4c45fd-690b-4111-9d15-93dc183b3e46") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 22:06:19.349163 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:19.349130    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8503b30-31d6-4454-a3fd-8bea4a527b04-original-pull-secret podName:b8503b30-31d6-4454-a3fd-8bea4a527b04 nodeName:}" failed. No retries permitted until 2026-10-30 22:06:20.349121345 +0000 UTC m=+4.102475569 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b8503b30-31d6-4454-a3fd-8bea4a527b04-original-pull-secret") pod "global-pull-secret-syncer-rjtc8" (UID: "b8503b30-31d6-4454-a3fd-8bea4a527b04") : object "kube-system"/"original-pull-secret" not registered
Oct 30 22:06:19.450308 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:19.450263    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8286n\" (UniqueName: \"kubernetes.io/projected/dbf6b14c-5576-4ae3-bbb3-b3201aa7cb7c-kube-api-access-8286n\") pod \"network-check-target-vs5pt\" (UID: \"dbf6b14c-5576-4ae3-bbb3-b3201aa7cb7c\") " pod="openshift-network-diagnostics/network-check-target-vs5pt"
Oct 30 22:06:19.460304 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:19.460274    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 22:06:19.460304 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:19.460309    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 22:06:19.460491 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:19.460332    2579 projected.go:196] Error preparing data for projected volume kube-api-access-8286n for pod openshift-network-diagnostics/network-check-target-vs5pt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 22:06:19.460491 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:19.460407    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dbf6b14c-5576-4ae3-bbb3-b3201aa7cb7c-kube-api-access-8286n podName:dbf6b14c-5576-4ae3-bbb3-b3201aa7cb7c nodeName:}" failed. No retries permitted until 2026-10-30 22:06:20.46038055 +0000 UTC m=+4.213734785 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-8286n" (UniqueName: "kubernetes.io/projected/dbf6b14c-5576-4ae3-bbb3-b3201aa7cb7c-kube-api-access-8286n") pod "network-check-target-vs5pt" (UID: "dbf6b14c-5576-4ae3-bbb3-b3201aa7cb7c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 22:06:19.693359 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:19.693101    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 22:01:18 +0000 UTC" deadline="2028-05-21 11:39:27.85860032 +0000 UTC"
Oct 30 22:06:19.693359 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:19.693155    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13645h33m8.165448666s"
Oct 30 22:06:19.756192 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:19.756164    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6hhw9"
Oct 30 22:06:19.756358 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:19.756327    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6hhw9" podUID="fe4c45fd-690b-4111-9d15-93dc183b3e46"
Oct 30 22:06:19.769853 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:19.769784    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-22c27" event={"ID":"15e3a459-855f-4181-825c-345e9b2e62e1","Type":"ContainerStarted","Data":"e0346fe028ed362ec3b464577562de2ae63214a8eaf5f062879fd52f0b3ca089"}
Oct 30 22:06:19.779370 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:19.779335    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mm655" event={"ID":"a3d96078-3ec8-40aa-855a-d1ba4acf2e74","Type":"ContainerStarted","Data":"328a6e8e630e43c5536d50c0c617888a400f575042f78833b718b6ab8188c58d"}
Oct 30 22:06:19.782565 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:19.782492    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dr9c2" event={"ID":"87f77489-c825-4f2c-a6ce-70e06f68f187","Type":"ContainerStarted","Data":"8fa2f930c7a689eefaa2d8987458a7101eb0907572b0cbafb8be10af6fb91d36"}
Oct 30 22:06:19.788311 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:19.788129    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-77.ec2.internal" event={"ID":"1929b84fce10dee760acfa96faf62b49","Type":"ContainerStarted","Data":"3746a9cd324d6d159937d32c5b63250f8da3be8ce7f3fedc055fab27b936a528"}
Oct 30 22:06:19.797281 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:19.796236    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-b74lk" event={"ID":"e9c20fdf-bea5-4d6c-b252-b29b08bfdf63","Type":"ContainerStarted","Data":"f93b6fcd91ff7bdca9eb5ca4eb05183ad2ac1e9b0da47e40d0bc8be3748fd2ec"}
Oct 30 22:06:19.799480 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:19.799383    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bbkdz" event={"ID":"0e128ead-13e5-41f3-9ae7-6f944b5f18ee","Type":"ContainerStarted","Data":"f47cd69ac5da515bb68e1c56a99c33e22b32498bf9de466ca92af4f71088dae2"}
Oct 30 22:06:19.801502 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:19.801478    2579 generic.go:356] "Generic (PLEG): container finished" podID="19ea3e9d10869a3d260d4dc45edca0a4" containerID="f042eb44796cb6b63e036a0c4f37226bee45acf9ec48c3dc9086ed7eaa81e479" exitCode=0
Oct 30 22:06:19.801630 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:19.801545    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-77.ec2.internal" event={"ID":"19ea3e9d10869a3d260d4dc45edca0a4","Type":"ContainerDied","Data":"f042eb44796cb6b63e036a0c4f37226bee45acf9ec48c3dc9086ed7eaa81e479"}
Oct 30 22:06:19.806811 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:19.806786    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l" event={"ID":"015f218d-bf1f-4e7e-bb5d-b805af003246","Type":"ContainerStarted","Data":"eaf4f0878c59e7dec85bba1fc9a00bd5871a054e5b82711075cba359d8242920"}
Oct 30 22:06:19.822926 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:19.822877    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vw5fg" event={"ID":"2e4eaf5f-8a83-4681-b040-65676affdf7e","Type":"ContainerStarted","Data":"285e3b8fbd1b072ec18318062c4f11065437f96d1131aaf0ec321e4aa459a467"}
Oct 30 22:06:19.829197 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:19.829175    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4knxq" event={"ID":"da9b75bf-8af2-450f-9a98-c265eea48282","Type":"ContainerStarted","Data":"faaad9c80829f8bc4d3a4972a649291ae22f1ad8792b1393f1aef56194fb3f56"}
Oct 30 22:06:19.850053 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:19.849980    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4s4vc" event={"ID":"7b50e755-6b0b-4d45-b7d3-c822127e6cb1","Type":"ContainerStarted","Data":"45efc55eedbfd6c64303396ae3d213a3b6cf4e9f53fc0a0bab6ab516227fe3c5"}
Oct 30 22:06:19.869183 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:19.868808    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-77.ec2.internal" podStartSLOduration=1.868791586 podStartE2EDuration="1.868791586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 22:06:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:19.828098001 +0000 UTC m=+3.581452242" watchObservedRunningTime="2026-10-30 22:06:19.868791586 +0000 UTC m=+3.622145827"
Oct 30 22:06:20.358502 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:20.358429    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b8503b30-31d6-4454-a3fd-8bea4a527b04-original-pull-secret\") pod \"global-pull-secret-syncer-rjtc8\" (UID: \"b8503b30-31d6-4454-a3fd-8bea4a527b04\") " pod="kube-system/global-pull-secret-syncer-rjtc8"
Oct 30 22:06:20.358664 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:20.358509    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe4c45fd-690b-4111-9d15-93dc183b3e46-metrics-certs\") pod \"network-metrics-daemon-6hhw9\" (UID: \"fe4c45fd-690b-4111-9d15-93dc183b3e46\") " pod="openshift-multus/network-metrics-daemon-6hhw9"
Oct 30 22:06:20.358664 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:20.358640    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 22:06:20.358775 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:20.358697    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe4c45fd-690b-4111-9d15-93dc183b3e46-metrics-certs podName:fe4c45fd-690b-4111-9d15-93dc183b3e46 nodeName:}" failed. No retries permitted until 2026-10-30 22:06:22.358678701 +0000 UTC m=+6.112032933 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fe4c45fd-690b-4111-9d15-93dc183b3e46-metrics-certs") pod "network-metrics-daemon-6hhw9" (UID: "fe4c45fd-690b-4111-9d15-93dc183b3e46") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 22:06:20.359093 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:20.359075    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 22:06:20.359172 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:20.359129    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8503b30-31d6-4454-a3fd-8bea4a527b04-original-pull-secret podName:b8503b30-31d6-4454-a3fd-8bea4a527b04 nodeName:}" failed. No retries permitted until 2026-10-30 22:06:22.359115197 +0000 UTC m=+6.112469431 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b8503b30-31d6-4454-a3fd-8bea4a527b04-original-pull-secret") pod "global-pull-secret-syncer-rjtc8" (UID: "b8503b30-31d6-4454-a3fd-8bea4a527b04") : object "kube-system"/"original-pull-secret" not registered
Oct 30 22:06:20.560390 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:20.560352    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8286n\" (UniqueName: \"kubernetes.io/projected/dbf6b14c-5576-4ae3-bbb3-b3201aa7cb7c-kube-api-access-8286n\") pod \"network-check-target-vs5pt\" (UID: \"dbf6b14c-5576-4ae3-bbb3-b3201aa7cb7c\") " pod="openshift-network-diagnostics/network-check-target-vs5pt"
Oct 30 22:06:20.560601 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:20.560551    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 22:06:20.560601 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:20.560570    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 22:06:20.560601 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:20.560592    2579 projected.go:196] Error preparing data for projected volume kube-api-access-8286n for pod openshift-network-diagnostics/network-check-target-vs5pt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 22:06:20.560750 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:20.560647    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dbf6b14c-5576-4ae3-bbb3-b3201aa7cb7c-kube-api-access-8286n podName:dbf6b14c-5576-4ae3-bbb3-b3201aa7cb7c nodeName:}" failed. No retries permitted until 2026-10-30 22:06:22.560630176 +0000 UTC m=+6.313984412 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-8286n" (UniqueName: "kubernetes.io/projected/dbf6b14c-5576-4ae3-bbb3-b3201aa7cb7c-kube-api-access-8286n") pod "network-check-target-vs5pt" (UID: "dbf6b14c-5576-4ae3-bbb3-b3201aa7cb7c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 22:06:20.693766 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:20.693719    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 22:01:18 +0000 UTC" deadline="2028-05-18 12:20:04.390092285 +0000 UTC"
Oct 30 22:06:20.693766 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:20.693757    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13574h13m43.696341339s"
Oct 30 22:06:20.757429 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:20.757402    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vs5pt"
Oct 30 22:06:20.757572 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:20.757524    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vs5pt" podUID="dbf6b14c-5576-4ae3-bbb3-b3201aa7cb7c"
Oct 30 22:06:20.757639 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:20.757609    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rjtc8"
Oct 30 22:06:20.757709 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:20.757684    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rjtc8" podUID="b8503b30-31d6-4454-a3fd-8bea4a527b04"
Oct 30 22:06:20.867839 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:20.867741    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-77.ec2.internal" event={"ID":"19ea3e9d10869a3d260d4dc45edca0a4","Type":"ContainerStarted","Data":"f1296a641251d40db80105088bb63befd770fa1cfd7d6d830455fb5566909dd3"}
Oct 30 22:06:21.755606 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:21.755542    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6hhw9"
Oct 30 22:06:21.756028 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:21.755721    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6hhw9" podUID="fe4c45fd-690b-4111-9d15-93dc183b3e46"
Oct 30 22:06:21.875143 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:21.875100    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-22c27" event={"ID":"15e3a459-855f-4181-825c-345e9b2e62e1","Type":"ContainerStarted","Data":"1405d0b5534998c510b75ae3dce5a799859b349d946be3c39d991cc54fb814e9"}
Oct 30 22:06:21.949702 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:21.949641    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-77.ec2.internal" podStartSLOduration=3.949620398 podStartE2EDuration="3.949620398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 22:06:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:20.908388593 +0000 UTC m=+4.661742833" watchObservedRunningTime="2026-10-30 22:06:21.949620398 +0000 UTC m=+5.702974660"
Oct 30 22:06:22.374410 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:22.374379    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe4c45fd-690b-4111-9d15-93dc183b3e46-metrics-certs\") pod \"network-metrics-daemon-6hhw9\" (UID: \"fe4c45fd-690b-4111-9d15-93dc183b3e46\") " pod="openshift-multus/network-metrics-daemon-6hhw9"
Oct 30 22:06:22.374593 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:22.374431    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b8503b30-31d6-4454-a3fd-8bea4a527b04-original-pull-secret\") pod \"global-pull-secret-syncer-rjtc8\" (UID: \"b8503b30-31d6-4454-a3fd-8bea4a527b04\") " pod="kube-system/global-pull-secret-syncer-rjtc8"
Oct 30 22:06:22.374593 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:22.374554    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 22:06:22.374708 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:22.374601    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8503b30-31d6-4454-a3fd-8bea4a527b04-original-pull-secret podName:b8503b30-31d6-4454-a3fd-8bea4a527b04 nodeName:}" failed. No retries permitted until 2026-10-30 22:06:26.374585814 +0000 UTC m=+10.127940045 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b8503b30-31d6-4454-a3fd-8bea4a527b04-original-pull-secret") pod "global-pull-secret-syncer-rjtc8" (UID: "b8503b30-31d6-4454-a3fd-8bea4a527b04") : object "kube-system"/"original-pull-secret" not registered
Oct 30 22:06:22.374708 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:22.374665    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 22:06:22.374708 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:22.374694    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe4c45fd-690b-4111-9d15-93dc183b3e46-metrics-certs podName:fe4c45fd-690b-4111-9d15-93dc183b3e46 nodeName:}" failed. No retries permitted until 2026-10-30 22:06:26.374686759 +0000 UTC m=+10.128040977 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fe4c45fd-690b-4111-9d15-93dc183b3e46-metrics-certs") pod "network-metrics-daemon-6hhw9" (UID: "fe4c45fd-690b-4111-9d15-93dc183b3e46") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 22:06:22.576127 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:22.576077    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8286n\" (UniqueName: \"kubernetes.io/projected/dbf6b14c-5576-4ae3-bbb3-b3201aa7cb7c-kube-api-access-8286n\") pod \"network-check-target-vs5pt\" (UID: \"dbf6b14c-5576-4ae3-bbb3-b3201aa7cb7c\") " pod="openshift-network-diagnostics/network-check-target-vs5pt"
Oct 30 22:06:22.576282 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:22.576233    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 22:06:22.576282 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:22.576251    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 22:06:22.576282 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:22.576267    2579 projected.go:196] Error preparing data for projected volume kube-api-access-8286n for pod openshift-network-diagnostics/network-check-target-vs5pt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 22:06:22.576439 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:22.576319    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dbf6b14c-5576-4ae3-bbb3-b3201aa7cb7c-kube-api-access-8286n podName:dbf6b14c-5576-4ae3-bbb3-b3201aa7cb7c nodeName:}" failed. No retries permitted until 2026-10-30 22:06:26.576301257 +0000 UTC m=+10.329655490 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-8286n" (UniqueName: "kubernetes.io/projected/dbf6b14c-5576-4ae3-bbb3-b3201aa7cb7c-kube-api-access-8286n") pod "network-check-target-vs5pt" (UID: "dbf6b14c-5576-4ae3-bbb3-b3201aa7cb7c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 22:06:22.759186 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:22.758261    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rjtc8"
Oct 30 22:06:22.759186 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:22.758395    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rjtc8" podUID="b8503b30-31d6-4454-a3fd-8bea4a527b04"
Oct 30 22:06:22.759186 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:22.758753    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vs5pt"
Oct 30 22:06:22.759186 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:22.758837    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vs5pt" podUID="dbf6b14c-5576-4ae3-bbb3-b3201aa7cb7c"
Oct 30 22:06:22.882143 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:22.882112    2579 generic.go:356] "Generic (PLEG): container finished" podID="15e3a459-855f-4181-825c-345e9b2e62e1" containerID="1405d0b5534998c510b75ae3dce5a799859b349d946be3c39d991cc54fb814e9" exitCode=0
Oct 30 22:06:22.882256 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:22.882166    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-22c27" event={"ID":"15e3a459-855f-4181-825c-345e9b2e62e1","Type":"ContainerDied","Data":"1405d0b5534998c510b75ae3dce5a799859b349d946be3c39d991cc54fb814e9"}
Oct 30 22:06:23.755939 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:23.755750    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6hhw9"
Oct 30 22:06:23.755939 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:23.755877    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6hhw9" podUID="fe4c45fd-690b-4111-9d15-93dc183b3e46"
Oct 30 22:06:23.892795 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:23.892750    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mm655" event={"ID":"a3d96078-3ec8-40aa-855a-d1ba4acf2e74","Type":"ContainerStarted","Data":"81d5f4e500b1dcaf049c5a2fadef4d645924ad6bb0b358e1199940c0243fa0cf"}
Oct 30 22:06:23.896772 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:23.896737    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-b74lk" event={"ID":"e9c20fdf-bea5-4d6c-b252-b29b08bfdf63","Type":"ContainerStarted","Data":"966f9d245f3a937da3ab9b20c8be894e41e8d0189926cea15e7ee879365c6675"}
Oct 30 22:06:23.975358 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:23.975305    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-b74lk" podStartSLOduration=3.393771665 podStartE2EDuration="7.975285337s" totalImagesPullingTime="4.581513672s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 22:06:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:23.97459423 +0000 UTC m=+7.727948475" watchObservedRunningTime="2026-10-30 22:06:23.975285337 +0000 UTC m=+7.728639578"
Oct 30 22:06:23.975526 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:23.975447    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-mm655" podStartSLOduration=4.28344247 podStartE2EDuration="7.97544001s" totalImagesPullingTime="3.69199754s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:06:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:23.934280093 +0000 UTC m=+7.687634334" watchObservedRunningTime="2026-10-30 22:06:23.97544001 +0000 UTC m=+7.728794251"
Oct 30 22:06:24.757701 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:24.757666    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rjtc8"
Oct 30 22:06:24.757868 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:24.757782    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rjtc8" podUID="b8503b30-31d6-4454-a3fd-8bea4a527b04"
Oct 30 22:06:24.758161 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:24.758142    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vs5pt"
Oct 30 22:06:24.758274 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:24.758225    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vs5pt" podUID="dbf6b14c-5576-4ae3-bbb3-b3201aa7cb7c"
Oct 30 22:06:24.907746 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:24.907312    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bbkdz" event={"ID":"0e128ead-13e5-41f3-9ae7-6f944b5f18ee","Type":"ContainerStarted","Data":"3af681ab97bd6be47be651cce5a7e2b6eb759e76f48a799fa92861ec48817dfb"}
Oct 30 22:06:25.756614 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:25.756252    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6hhw9"
Oct 30 22:06:25.756614 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:25.756350    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6hhw9" podUID="fe4c45fd-690b-4111-9d15-93dc183b3e46"
Oct 30 22:06:25.872746 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:25.872712    2579 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 22:06:25.913111 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:25.913082    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bbkdz" event={"ID":"0e128ead-13e5-41f3-9ae7-6f944b5f18ee","Type":"ContainerStarted","Data":"18b678a2ff11faabccb1f8a548d6dea92543824f3712e911bdc0cd02a0590f81"}
Oct 30 22:06:26.407571 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:26.407529    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe4c45fd-690b-4111-9d15-93dc183b3e46-metrics-certs\") pod \"network-metrics-daemon-6hhw9\" (UID: \"fe4c45fd-690b-4111-9d15-93dc183b3e46\") " pod="openshift-multus/network-metrics-daemon-6hhw9"
Oct 30 22:06:26.407734 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:26.407604    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b8503b30-31d6-4454-a3fd-8bea4a527b04-original-pull-secret\") pod \"global-pull-secret-syncer-rjtc8\" (UID: \"b8503b30-31d6-4454-a3fd-8bea4a527b04\") " pod="kube-system/global-pull-secret-syncer-rjtc8"
Oct 30 22:06:26.407790 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:26.407769    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 22:06:26.407834 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:26.407826    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8503b30-31d6-4454-a3fd-8bea4a527b04-original-pull-secret podName:b8503b30-31d6-4454-a3fd-8bea4a527b04 nodeName:}" failed. No retries permitted until 2026-10-30 22:06:34.407807527 +0000 UTC m=+18.161161762 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b8503b30-31d6-4454-a3fd-8bea4a527b04-original-pull-secret") pod "global-pull-secret-syncer-rjtc8" (UID: "b8503b30-31d6-4454-a3fd-8bea4a527b04") : object "kube-system"/"original-pull-secret" not registered
Oct 30 22:06:26.408210 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:26.408178    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 22:06:26.408280 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:26.408262    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe4c45fd-690b-4111-9d15-93dc183b3e46-metrics-certs podName:fe4c45fd-690b-4111-9d15-93dc183b3e46 nodeName:}" failed. No retries permitted until 2026-10-30 22:06:34.408244361 +0000 UTC m=+18.161598600 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fe4c45fd-690b-4111-9d15-93dc183b3e46-metrics-certs") pod "network-metrics-daemon-6hhw9" (UID: "fe4c45fd-690b-4111-9d15-93dc183b3e46") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 22:06:26.611602 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:26.610899    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8286n\" (UniqueName: \"kubernetes.io/projected/dbf6b14c-5576-4ae3-bbb3-b3201aa7cb7c-kube-api-access-8286n\") pod \"network-check-target-vs5pt\" (UID: \"dbf6b14c-5576-4ae3-bbb3-b3201aa7cb7c\") " pod="openshift-network-diagnostics/network-check-target-vs5pt"
Oct 30 22:06:26.611602 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:26.611119    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 22:06:26.611602 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:26.611139    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 22:06:26.611602 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:26.611155    2579 projected.go:196] Error preparing data for projected volume kube-api-access-8286n for pod openshift-network-diagnostics/network-check-target-vs5pt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 22:06:26.611602 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:26.611214    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dbf6b14c-5576-4ae3-bbb3-b3201aa7cb7c-kube-api-access-8286n podName:dbf6b14c-5576-4ae3-bbb3-b3201aa7cb7c nodeName:}" failed. No retries permitted until 2026-10-30 22:06:34.611195967 +0000 UTC m=+18.364550199 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-8286n" (UniqueName: "kubernetes.io/projected/dbf6b14c-5576-4ae3-bbb3-b3201aa7cb7c-kube-api-access-8286n") pod "network-check-target-vs5pt" (UID: "dbf6b14c-5576-4ae3-bbb3-b3201aa7cb7c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 22:06:26.688373 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:26.687459    2579 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T22:06:25.872728814Z","UUID":"d5ff1849-3868-4024-aa24-97819e5385b3","Handler":null,"Name":"","Endpoint":""}
Oct 30 22:06:26.694229 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:26.693970    2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 22:06:26.694229 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:26.693997    2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 22:06:26.757173 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:26.757144    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rjtc8"
Oct 30 22:06:26.757173 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:26.757163    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vs5pt"
Oct 30 22:06:26.757329 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:26.757259    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vs5pt" podUID="dbf6b14c-5576-4ae3-bbb3-b3201aa7cb7c"
Oct 30 22:06:26.758407 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:26.758352    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rjtc8" podUID="b8503b30-31d6-4454-a3fd-8bea4a527b04"
Oct 30 22:06:26.917179 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:26.916654    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4knxq" event={"ID":"da9b75bf-8af2-450f-9a98-c265eea48282","Type":"ContainerStarted","Data":"44b837a2a9523e8050a83285a486b8cdacd18a4f9a1a012433bdadd540945d51"}
Oct 30 22:06:26.919754 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:26.919351    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4s4vc" event={"ID":"7b50e755-6b0b-4d45-b7d3-c822127e6cb1","Type":"ContainerStarted","Data":"812c12ca6a874552a8f105f60c2139c48c48e8daf42096106bb3de0b0d720133"}
Oct 30 22:06:27.007678 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:27.005560    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-4s4vc" podStartSLOduration=3.764865943 podStartE2EDuration="11.005544382s" totalImagesPullingTime="7.240678439s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 22:06:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:27.005209521 +0000 UTC m=+10.758563762" watchObservedRunningTime="2026-10-30 22:06:27.005544382 +0000 UTC m=+10.758898623"
Oct 30 22:06:27.007678 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:27.006192    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-4knxq" podStartSLOduration=3.304169638 podStartE2EDuration="11.006181763s" totalImagesPullingTime="7.702012125s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 22:06:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:26.958938731 +0000 UTC m=+10.712292971" watchObservedRunningTime="2026-10-30 22:06:27.006181763 +0000 UTC m=+10.759536009"
Oct 30 22:06:27.756330 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:27.756124    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6hhw9"
Oct 30 22:06:27.756580 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:27.756438    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6hhw9" podUID="fe4c45fd-690b-4111-9d15-93dc183b3e46"
Oct 30 22:06:27.923474 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:27.923435    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bbkdz" event={"ID":"0e128ead-13e5-41f3-9ae7-6f944b5f18ee","Type":"ContainerStarted","Data":"7f7472dff4e2f4ce1c1de1f3f1941c048ef91c3fb42d25668bb77f7097afd5a7"}
Oct 30 22:06:27.925787 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:27.925757    2579 generic.go:356] "Generic (PLEG): container finished" podID="15e3a459-855f-4181-825c-345e9b2e62e1" containerID="da8e5996fea0205110b62b279a7a2b5674fb4bedfd8d9c2a0ded60ba1e3b1446" exitCode=0
Oct 30 22:06:27.925913 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:27.925858    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-22c27" event={"ID":"15e3a459-855f-4181-825c-345e9b2e62e1","Type":"ContainerDied","Data":"da8e5996fea0205110b62b279a7a2b5674fb4bedfd8d9c2a0ded60ba1e3b1446"}
Oct 30 22:06:27.966142 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:27.966099    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bbkdz" podStartSLOduration=4.247932983 podStartE2EDuration="11.966086469s" totalImagesPullingTime="7.718153486s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 22:06:16 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:27.965803648 +0000 UTC m=+11.719157888" watchObservedRunningTime="2026-10-30 22:06:27.966086469 +0000 UTC m=+11.719440710"
Oct 30 22:06:28.756195 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:28.756146    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vs5pt"
Oct 30 22:06:28.756323 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:28.756295    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vs5pt" podUID="dbf6b14c-5576-4ae3-bbb3-b3201aa7cb7c"
Oct 30 22:06:28.756371 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:28.756337    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rjtc8"
Oct 30 22:06:28.756423 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:28.756408    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rjtc8" podUID="b8503b30-31d6-4454-a3fd-8bea4a527b04"
Oct 30 22:06:28.818558 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:28.818534    2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-mm655"
Oct 30 22:06:28.819248 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:28.819230    2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-mm655"
Oct 30 22:06:28.929604 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:28.929565    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vw5fg" event={"ID":"2e4eaf5f-8a83-4681-b040-65676affdf7e","Type":"ContainerStarted","Data":"a2ee3988dce541df70ed441b6ab921234a97e6c7ff7990cdbd165d23891fb700"}
Oct 30 22:06:28.931843 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:28.931818    2579 generic.go:356] "Generic (PLEG): container finished" podID="15e3a459-855f-4181-825c-345e9b2e62e1" containerID="451659c940d92507abf61f860658709d783592769d5f73ee9f4633e9a81dbed4" exitCode=0
Oct 30 22:06:28.931953 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:28.931892    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-22c27" event={"ID":"15e3a459-855f-4181-825c-345e9b2e62e1","Type":"ContainerDied","Data":"451659c940d92507abf61f860658709d783592769d5f73ee9f4633e9a81dbed4"}
Oct 30 22:06:28.933491 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:28.933467    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dr9c2" event={"ID":"87f77489-c825-4f2c-a6ce-70e06f68f187","Type":"ContainerStarted","Data":"738d72edf25cfc3ce43c38c1045038a2d2cfc5d83b1a6147d42ae8ae0eb25438"}
Oct 30 22:06:28.933716 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:28.933615    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-mm655"
Oct 30 22:06:28.934074 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:28.934058    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-mm655"
Oct 30 22:06:28.971661 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:28.971615    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-vw5fg" podStartSLOduration=5.691464517 podStartE2EDuration="12.971598024s" totalImagesPullingTime="7.280133507s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 22:06:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:28.970871965 +0000 UTC m=+12.724226204" watchObservedRunningTime="2026-10-30 22:06:28.971598024 +0000 UTC m=+12.724952265"
Oct 30 22:06:29.012751 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:29.012671    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-dr9c2" podStartSLOduration=3.4999228589999998 podStartE2EDuration="13.012658939s" totalImagesPullingTime="9.51273608s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 22:06:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:29.012424657 +0000 UTC m=+12.765778891" watchObservedRunningTime="2026-10-30 22:06:29.012658939 +0000 UTC m=+12.766013177"
Oct 30 22:06:29.755684 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:29.755652    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6hhw9"
Oct 30 22:06:29.755847 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:29.755779    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6hhw9" podUID="fe4c45fd-690b-4111-9d15-93dc183b3e46"
Oct 30 22:06:29.937846 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:29.937809    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-22c27" event={"ID":"15e3a459-855f-4181-825c-345e9b2e62e1","Type":"ContainerStarted","Data":"126d7083d31d188d3e448a6fd2bfc2eb9c8e63e854d43dcc9fb52cbb0d431239"}
Oct 30 22:06:30.755424 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:30.755391    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vs5pt"
Oct 30 22:06:30.755424 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:30.755429    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rjtc8"
Oct 30 22:06:30.755659 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:30.755522    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vs5pt" podUID="dbf6b14c-5576-4ae3-bbb3-b3201aa7cb7c"
Oct 30 22:06:30.755720 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:30.755686    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rjtc8" podUID="b8503b30-31d6-4454-a3fd-8bea4a527b04"
Oct 30 22:06:30.942468 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:30.942433    2579 generic.go:356] "Generic (PLEG): container finished" podID="15e3a459-855f-4181-825c-345e9b2e62e1" containerID="126d7083d31d188d3e448a6fd2bfc2eb9c8e63e854d43dcc9fb52cbb0d431239" exitCode=0
Oct 30 22:06:30.943065 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:30.942513    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-22c27" event={"ID":"15e3a459-855f-4181-825c-345e9b2e62e1","Type":"ContainerDied","Data":"126d7083d31d188d3e448a6fd2bfc2eb9c8e63e854d43dcc9fb52cbb0d431239"}
Oct 30 22:06:31.756306 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:31.756102    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6hhw9"
Oct 30 22:06:31.756465 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:31.756412    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6hhw9" podUID="fe4c45fd-690b-4111-9d15-93dc183b3e46"
Oct 30 22:06:32.755993 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:32.755955    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rjtc8"
Oct 30 22:06:32.756468 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:32.756093    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rjtc8" podUID="b8503b30-31d6-4454-a3fd-8bea4a527b04"
Oct 30 22:06:32.756468 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:32.756145    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vs5pt"
Oct 30 22:06:32.756468 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:32.756232    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vs5pt" podUID="dbf6b14c-5576-4ae3-bbb3-b3201aa7cb7c"
Oct 30 22:06:33.755753 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:33.755722    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6hhw9"
Oct 30 22:06:33.755957 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:33.755858    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6hhw9" podUID="fe4c45fd-690b-4111-9d15-93dc183b3e46"
Oct 30 22:06:33.951481 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:33.951444    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jcf8l_015f218d-bf1f-4e7e-bb5d-b805af003246/ovn-acl-logging/0.log"
Oct 30 22:06:33.951875 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:33.951767    2579 generic.go:356] "Generic (PLEG): container finished" podID="015f218d-bf1f-4e7e-bb5d-b805af003246" containerID="4d528146f7ae00a132df470b558b19deb09cbf4fb7b12143466a4aba4409814f" exitCode=1
Oct 30 22:06:33.951875 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:33.951841    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l" event={"ID":"015f218d-bf1f-4e7e-bb5d-b805af003246","Type":"ContainerStarted","Data":"d33ab18efac561c694cbf96c6caaf8d1e89b2c95ca4756f38153ac98d13f851b"}
Oct 30 22:06:33.951997 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:33.951928    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l" event={"ID":"015f218d-bf1f-4e7e-bb5d-b805af003246","Type":"ContainerStarted","Data":"578602373efc908948cc378c439427b39b79301c280766e61ab0d37c6e71c0a5"}
Oct 30 22:06:33.951997 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:33.951984    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l" event={"ID":"015f218d-bf1f-4e7e-bb5d-b805af003246","Type":"ContainerStarted","Data":"d13d448db1353c73bb8c1dffff88587f74def6224f3fc2a1508396d41df20ac0"}
Oct 30 22:06:33.952178 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:33.952032    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l" event={"ID":"015f218d-bf1f-4e7e-bb5d-b805af003246","Type":"ContainerStarted","Data":"1fd208d97d2b9c662d5e700ee72223d6f11635665e2531617317cc6d629eebf9"}
Oct 30 22:06:33.952178 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:33.952097    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l" event={"ID":"015f218d-bf1f-4e7e-bb5d-b805af003246","Type":"ContainerDied","Data":"4d528146f7ae00a132df470b558b19deb09cbf4fb7b12143466a4aba4409814f"}
Oct 30 22:06:33.952178 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:33.952147    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l" event={"ID":"015f218d-bf1f-4e7e-bb5d-b805af003246","Type":"ContainerStarted","Data":"e3e18c9191edb5d9c8a928c974cba60224c0532e572c6d6786f76efefee383b8"}
Oct 30 22:06:33.956262 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:33.955349    2579 generic.go:356] "Generic (PLEG): container finished" podID="15e3a459-855f-4181-825c-345e9b2e62e1" containerID="5063ef5a1b132a05514d139322ab1915957775308c5b0e42b612db3072b4e958" exitCode=0
Oct 30 22:06:33.956262 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:33.955405    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-22c27" event={"ID":"15e3a459-855f-4181-825c-345e9b2e62e1","Type":"ContainerDied","Data":"5063ef5a1b132a05514d139322ab1915957775308c5b0e42b612db3072b4e958"}
Oct 30 22:06:34.467293 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:34.467252    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b8503b30-31d6-4454-a3fd-8bea4a527b04-original-pull-secret\") pod \"global-pull-secret-syncer-rjtc8\" (UID: \"b8503b30-31d6-4454-a3fd-8bea4a527b04\") " pod="kube-system/global-pull-secret-syncer-rjtc8"
Oct 30 22:06:34.467444 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:34.467343    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe4c45fd-690b-4111-9d15-93dc183b3e46-metrics-certs\") pod \"network-metrics-daemon-6hhw9\" (UID: \"fe4c45fd-690b-4111-9d15-93dc183b3e46\") " pod="openshift-multus/network-metrics-daemon-6hhw9"
Oct 30 22:06:34.467444 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:34.467404    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 22:06:34.467568 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:34.467450    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 22:06:34.467568 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:34.467467    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8503b30-31d6-4454-a3fd-8bea4a527b04-original-pull-secret podName:b8503b30-31d6-4454-a3fd-8bea4a527b04 nodeName:}" failed. No retries permitted until 2026-10-30 22:06:50.467450833 +0000 UTC m=+34.220805093 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b8503b30-31d6-4454-a3fd-8bea4a527b04-original-pull-secret") pod "global-pull-secret-syncer-rjtc8" (UID: "b8503b30-31d6-4454-a3fd-8bea4a527b04") : object "kube-system"/"original-pull-secret" not registered
Oct 30 22:06:34.467568 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:34.467503    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe4c45fd-690b-4111-9d15-93dc183b3e46-metrics-certs podName:fe4c45fd-690b-4111-9d15-93dc183b3e46 nodeName:}" failed. No retries permitted until 2026-10-30 22:06:50.46748653 +0000 UTC m=+34.220840804 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fe4c45fd-690b-4111-9d15-93dc183b3e46-metrics-certs") pod "network-metrics-daemon-6hhw9" (UID: "fe4c45fd-690b-4111-9d15-93dc183b3e46") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 22:06:34.668672 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:34.668635    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8286n\" (UniqueName: \"kubernetes.io/projected/dbf6b14c-5576-4ae3-bbb3-b3201aa7cb7c-kube-api-access-8286n\") pod \"network-check-target-vs5pt\" (UID: \"dbf6b14c-5576-4ae3-bbb3-b3201aa7cb7c\") " pod="openshift-network-diagnostics/network-check-target-vs5pt"
Oct 30 22:06:34.668808 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:34.668749    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 22:06:34.668808 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:34.668763    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 22:06:34.668808 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:34.668772    2579 projected.go:196] Error preparing data for projected volume kube-api-access-8286n for pod openshift-network-diagnostics/network-check-target-vs5pt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 22:06:34.668927 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:34.668812    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dbf6b14c-5576-4ae3-bbb3-b3201aa7cb7c-kube-api-access-8286n podName:dbf6b14c-5576-4ae3-bbb3-b3201aa7cb7c nodeName:}" failed. No retries permitted until 2026-10-30 22:06:50.668800181 +0000 UTC m=+34.422154398 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-8286n" (UniqueName: "kubernetes.io/projected/dbf6b14c-5576-4ae3-bbb3-b3201aa7cb7c-kube-api-access-8286n") pod "network-check-target-vs5pt" (UID: "dbf6b14c-5576-4ae3-bbb3-b3201aa7cb7c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 22:06:34.756055 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:34.755977    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vs5pt"
Oct 30 22:06:34.756055 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:34.756012    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rjtc8"
Oct 30 22:06:34.756242 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:34.756128    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vs5pt" podUID="dbf6b14c-5576-4ae3-bbb3-b3201aa7cb7c"
Oct 30 22:06:34.756299 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:34.756244    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rjtc8" podUID="b8503b30-31d6-4454-a3fd-8bea4a527b04"
Oct 30 22:06:34.963620 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:34.963591    2579 generic.go:356] "Generic (PLEG): container finished" podID="15e3a459-855f-4181-825c-345e9b2e62e1" containerID="7d09f456e8c9e6451509bbc44aacca9e936b21bd4a2b31aa7b67acf2e1ba6cb5" exitCode=0
Oct 30 22:06:34.964215 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:34.963643    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-22c27" event={"ID":"15e3a459-855f-4181-825c-345e9b2e62e1","Type":"ContainerDied","Data":"7d09f456e8c9e6451509bbc44aacca9e936b21bd4a2b31aa7b67acf2e1ba6cb5"}
Oct 30 22:06:35.755446 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:35.755386    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6hhw9"
Oct 30 22:06:35.755569 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:35.755482    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6hhw9" podUID="fe4c45fd-690b-4111-9d15-93dc183b3e46"
Oct 30 22:06:35.968601 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:35.968555    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-22c27" event={"ID":"15e3a459-855f-4181-825c-345e9b2e62e1","Type":"ContainerStarted","Data":"a0cad7033dfc246a82638bc483a9bbcb56ada187d352c0d822e5fdae6358ea11"}
Oct 30 22:06:35.971201 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:35.971180    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jcf8l_015f218d-bf1f-4e7e-bb5d-b805af003246/ovn-acl-logging/0.log"
Oct 30 22:06:35.971528 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:35.971506    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l" event={"ID":"015f218d-bf1f-4e7e-bb5d-b805af003246","Type":"ContainerStarted","Data":"c12f7ad1a632f8a46db91a3e92a31212d3223d49f8f171b8cfc071092d2ae56c"}
Oct 30 22:06:36.044125 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:36.044035    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-22c27" podStartSLOduration=9.754481207 podStartE2EDuration="20.044021752s" totalImagesPullingTime="10.289540545s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 22:06:16 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:36.04351935 +0000 UTC m=+19.796873602" watchObservedRunningTime="2026-10-30 22:06:36.044021752 +0000 UTC m=+19.797375991"
Oct 30 22:06:36.756150 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:36.756123    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rjtc8"
Oct 30 22:06:36.756298 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:36.756198    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rjtc8" podUID="b8503b30-31d6-4454-a3fd-8bea4a527b04"
Oct 30 22:06:36.756343 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:36.756300    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vs5pt"
Oct 30 22:06:36.756421 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:36.756400    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vs5pt" podUID="dbf6b14c-5576-4ae3-bbb3-b3201aa7cb7c"
Oct 30 22:06:37.755747 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:37.755716    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6hhw9"
Oct 30 22:06:37.756296 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:37.755830    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6hhw9" podUID="fe4c45fd-690b-4111-9d15-93dc183b3e46"
Oct 30 22:06:37.979237 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:37.979211    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jcf8l_015f218d-bf1f-4e7e-bb5d-b805af003246/ovn-acl-logging/0.log"
Oct 30 22:06:37.979609 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:37.979583    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l" event={"ID":"015f218d-bf1f-4e7e-bb5d-b805af003246","Type":"ContainerStarted","Data":"ebd518069b779d87af008174fab62e83920e88b00c5af2512a314e9470c8307b"}
Oct 30 22:06:37.979863 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:37.979849    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l"
Oct 30 22:06:37.980094 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:37.980080    2579 scope.go:122] "RemoveContainer" containerID="4d528146f7ae00a132df470b558b19deb09cbf4fb7b12143466a4aba4409814f"
Oct 30 22:06:37.996220 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:37.996201    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l"
Oct 30 22:06:38.755277 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:38.755237    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rjtc8"
Oct 30 22:06:38.755277 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:38.755263    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vs5pt"
Oct 30 22:06:38.755491 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:38.755370    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rjtc8" podUID="b8503b30-31d6-4454-a3fd-8bea4a527b04"
Oct 30 22:06:38.755532 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:38.755496    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vs5pt" podUID="dbf6b14c-5576-4ae3-bbb3-b3201aa7cb7c"
Oct 30 22:06:38.985716 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:38.985689    2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jcf8l_015f218d-bf1f-4e7e-bb5d-b805af003246/ovn-acl-logging/0.log"
Oct 30 22:06:38.986147 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:38.985998    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l" event={"ID":"015f218d-bf1f-4e7e-bb5d-b805af003246","Type":"ContainerStarted","Data":"ff8ea2f87b67a51749b4ffc5eec0624ebdf2c9f6733153034fac4e8304da3241"}
Oct 30 22:06:38.986248 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:38.986232    2579 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 22:06:38.986457 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:38.986439    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l"
Oct 30 22:06:39.000429 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:39.000406    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l"
Oct 30 22:06:39.081149 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:39.081073    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l" podStartSLOduration=9.144748893 podStartE2EDuration="23.081062892s" totalImagesPullingTime="13.936313999s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 22:06:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:39.067940502 +0000 UTC m=+22.821294743" watchObservedRunningTime="2026-10-30 22:06:39.081062892 +0000 UTC m=+22.834417132"
Oct 30 22:06:39.628208 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:39.628180    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-vs5pt"]
Oct 30 22:06:39.628377 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:39.628317    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vs5pt"
Oct 30 22:06:39.628442 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:39.628395    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vs5pt" podUID="dbf6b14c-5576-4ae3-bbb3-b3201aa7cb7c"
Oct 30 22:06:39.632140 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:39.632117    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-rjtc8"]
Oct 30 22:06:39.632260 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:39.632162    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6hhw9"]
Oct 30 22:06:39.632260 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:39.632229    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6hhw9"
Oct 30 22:06:39.632348 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:39.632328    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6hhw9" podUID="fe4c45fd-690b-4111-9d15-93dc183b3e46"
Oct 30 22:06:39.632427 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:39.632412    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rjtc8"
Oct 30 22:06:39.632510 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:39.632493    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rjtc8" podUID="b8503b30-31d6-4454-a3fd-8bea4a527b04"
Oct 30 22:06:39.988205 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:39.988138    2579 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 22:06:40.756237 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:40.756202    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rjtc8"
Oct 30 22:06:40.756499 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:40.756317    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rjtc8" podUID="b8503b30-31d6-4454-a3fd-8bea4a527b04"
Oct 30 22:06:40.990029 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:40.990001    2579 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 22:06:41.755547 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:41.755519    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6hhw9"
Oct 30 22:06:41.755656 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:41.755529    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vs5pt"
Oct 30 22:06:41.755693 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:41.755647    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6hhw9" podUID="fe4c45fd-690b-4111-9d15-93dc183b3e46"
Oct 30 22:06:41.755693 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:41.755684    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vs5pt" podUID="dbf6b14c-5576-4ae3-bbb3-b3201aa7cb7c"
Oct 30 22:06:42.755506 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:42.755478    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rjtc8"
Oct 30 22:06:42.755864 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:42.755586    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rjtc8" podUID="b8503b30-31d6-4454-a3fd-8bea4a527b04"
Oct 30 22:06:43.755602 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:43.755568    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6hhw9"
Oct 30 22:06:43.755993 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:43.755720    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vs5pt"
Oct 30 22:06:43.755993 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:43.755735    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6hhw9" podUID="fe4c45fd-690b-4111-9d15-93dc183b3e46"
Oct 30 22:06:43.755993 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:43.755804    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vs5pt" podUID="dbf6b14c-5576-4ae3-bbb3-b3201aa7cb7c"
Oct 30 22:06:44.421217 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:44.421177    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l"
Oct 30 22:06:44.421405 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:44.421393    2579 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 22:06:44.431127 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:44.431089    2579 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l" podUID="015f218d-bf1f-4e7e-bb5d-b805af003246" containerName="ovnkube-controller" probeResult="failure" output=""
Oct 30 22:06:44.438940 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:44.438915    2579 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l" podUID="015f218d-bf1f-4e7e-bb5d-b805af003246" containerName="ovnkube-controller" probeResult="failure" output=""
Oct 30 22:06:44.755636 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:44.755437    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rjtc8"
Oct 30 22:06:44.755962 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:44.755671    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rjtc8" podUID="b8503b30-31d6-4454-a3fd-8bea4a527b04"
Oct 30 22:06:45.756229 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:45.756200    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6hhw9"
Oct 30 22:06:45.756569 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:45.756301    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6hhw9" podUID="fe4c45fd-690b-4111-9d15-93dc183b3e46"
Oct 30 22:06:45.756569 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:45.756371    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vs5pt"
Oct 30 22:06:45.756569 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:45.756442    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vs5pt" podUID="dbf6b14c-5576-4ae3-bbb3-b3201aa7cb7c"
Oct 30 22:06:46.755941 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:46.755911    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rjtc8"
Oct 30 22:06:46.756070 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:46.756001    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rjtc8" podUID="b8503b30-31d6-4454-a3fd-8bea4a527b04"
Oct 30 22:06:46.968809 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:46.968752    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-77.ec2.internal" event="NodeReady"
Oct 30 22:06:46.969131 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:46.968833    2579 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 30 22:06:47.024111 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:47.024082    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-c749j"]
Oct 30 22:06:47.026326 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:47.026305    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-c749j"
Oct 30 22:06:47.026986 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:47.026966    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-wp8vp"]
Oct 30 22:06:47.028561 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:47.028545    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-wp8vp"
Oct 30 22:06:47.042684 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:47.042666    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 22:06:47.042775 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:47.042760    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 22:06:47.042987 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:47.042976    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-z6gtt\""
Oct 30 22:06:47.044313 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:47.044295    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 22:06:47.044460 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:47.044446    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 22:06:47.044526 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:47.044513    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 22:06:47.044588 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:47.044578    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-htsvp\""
Oct 30 22:06:47.050083 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:47.050065    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-c749j"]
Oct 30 22:06:47.056980 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:47.056959    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-wp8vp"]
Oct 30 22:06:47.163452 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:47.163428    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9ea63ef4-6b84-46e6-a753-bca60280734a-metrics-tls\") pod \"dns-default-c749j\" (UID: \"9ea63ef4-6b84-46e6-a753-bca60280734a\") " pod="openshift-dns/dns-default-c749j"
Oct 30 22:06:47.163534 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:47.163463    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckg9z\" (UniqueName: \"kubernetes.io/projected/9ea63ef4-6b84-46e6-a753-bca60280734a-kube-api-access-ckg9z\") pod \"dns-default-c749j\" (UID: \"9ea63ef4-6b84-46e6-a753-bca60280734a\") " pod="openshift-dns/dns-default-c749j"
Oct 30 22:06:47.163534 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:47.163489    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ea63ef4-6b84-46e6-a753-bca60280734a-tmp-dir\") pod \"dns-default-c749j\" (UID: \"9ea63ef4-6b84-46e6-a753-bca60280734a\") " pod="openshift-dns/dns-default-c749j"
Oct 30 22:06:47.163652 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:47.163628    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9ea63ef4-6b84-46e6-a753-bca60280734a-config-volume\") pod \"dns-default-c749j\" (UID: \"9ea63ef4-6b84-46e6-a753-bca60280734a\") " pod="openshift-dns/dns-default-c749j"
Oct 30 22:06:47.163686 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:47.163676    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/98075172-cd56-41f2-8203-fc2639e95df9-cert\") pod \"ingress-canary-wp8vp\" (UID: \"98075172-cd56-41f2-8203-fc2639e95df9\") " pod="openshift-ingress-canary/ingress-canary-wp8vp"
Oct 30 22:06:47.163727 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:47.163693    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fngn\" (UniqueName: \"kubernetes.io/projected/98075172-cd56-41f2-8203-fc2639e95df9-kube-api-access-8fngn\") pod \"ingress-canary-wp8vp\" (UID: \"98075172-cd56-41f2-8203-fc2639e95df9\") " pod="openshift-ingress-canary/ingress-canary-wp8vp"
Oct 30 22:06:47.264275 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:47.264218    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9ea63ef4-6b84-46e6-a753-bca60280734a-config-volume\") pod \"dns-default-c749j\" (UID: \"9ea63ef4-6b84-46e6-a753-bca60280734a\") " pod="openshift-dns/dns-default-c749j"
Oct 30 22:06:47.264275 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:47.264261    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/98075172-cd56-41f2-8203-fc2639e95df9-cert\") pod \"ingress-canary-wp8vp\" (UID: \"98075172-cd56-41f2-8203-fc2639e95df9\") " pod="openshift-ingress-canary/ingress-canary-wp8vp"
Oct 30 22:06:47.264428 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:47.264280    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fngn\" (UniqueName: \"kubernetes.io/projected/98075172-cd56-41f2-8203-fc2639e95df9-kube-api-access-8fngn\") pod \"ingress-canary-wp8vp\" (UID: \"98075172-cd56-41f2-8203-fc2639e95df9\") " pod="openshift-ingress-canary/ingress-canary-wp8vp"
Oct 30 22:06:47.264428 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:47.264349    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9ea63ef4-6b84-46e6-a753-bca60280734a-metrics-tls\") pod \"dns-default-c749j\" (UID: \"9ea63ef4-6b84-46e6-a753-bca60280734a\") " pod="openshift-dns/dns-default-c749j"
Oct 30 22:06:47.264428 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:47.264378    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckg9z\" (UniqueName: \"kubernetes.io/projected/9ea63ef4-6b84-46e6-a753-bca60280734a-kube-api-access-ckg9z\") pod \"dns-default-c749j\" (UID: \"9ea63ef4-6b84-46e6-a753-bca60280734a\") " pod="openshift-dns/dns-default-c749j"
Oct 30 22:06:47.264428 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:47.264398    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ea63ef4-6b84-46e6-a753-bca60280734a-tmp-dir\") pod \"dns-default-c749j\" (UID: \"9ea63ef4-6b84-46e6-a753-bca60280734a\") " pod="openshift-dns/dns-default-c749j"
Oct 30 22:06:47.264651 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:47.264512    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 22:06:47.264651 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:47.264572    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ea63ef4-6b84-46e6-a753-bca60280734a-metrics-tls podName:9ea63ef4-6b84-46e6-a753-bca60280734a nodeName:}" failed. No retries permitted until 2026-10-30 22:06:47.764553869 +0000 UTC m=+31.517908109 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9ea63ef4-6b84-46e6-a753-bca60280734a-metrics-tls") pod "dns-default-c749j" (UID: "9ea63ef4-6b84-46e6-a753-bca60280734a") : secret "dns-default-metrics-tls" not found
Oct 30 22:06:47.264767 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:47.264699    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ea63ef4-6b84-46e6-a753-bca60280734a-tmp-dir\") pod \"dns-default-c749j\" (UID: \"9ea63ef4-6b84-46e6-a753-bca60280734a\") " pod="openshift-dns/dns-default-c749j"
Oct 30 22:06:47.264962 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:47.264936    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9ea63ef4-6b84-46e6-a753-bca60280734a-config-volume\") pod \"dns-default-c749j\" (UID: \"9ea63ef4-6b84-46e6-a753-bca60280734a\") " pod="openshift-dns/dns-default-c749j"
Oct 30 22:06:47.268587 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:47.268561    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/98075172-cd56-41f2-8203-fc2639e95df9-cert\") pod \"ingress-canary-wp8vp\" (UID: \"98075172-cd56-41f2-8203-fc2639e95df9\") " pod="openshift-ingress-canary/ingress-canary-wp8vp"
Oct 30 22:06:47.285555 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:47.285523    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckg9z\" (UniqueName: \"kubernetes.io/projected/9ea63ef4-6b84-46e6-a753-bca60280734a-kube-api-access-ckg9z\") pod \"dns-default-c749j\" (UID: \"9ea63ef4-6b84-46e6-a753-bca60280734a\") " pod="openshift-dns/dns-default-c749j"
Oct 30 22:06:47.285632 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:47.285606    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fngn\" (UniqueName: \"kubernetes.io/projected/98075172-cd56-41f2-8203-fc2639e95df9-kube-api-access-8fngn\") pod \"ingress-canary-wp8vp\" (UID: \"98075172-cd56-41f2-8203-fc2639e95df9\") " pod="openshift-ingress-canary/ingress-canary-wp8vp"
Oct 30 22:06:47.341682 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:47.341662    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-wp8vp"
Oct 30 22:06:47.641995 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:47.641966    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-wp8vp"]
Oct 30 22:06:47.645182 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:47.645155    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98075172_cd56_41f2_8203_fc2639e95df9.slice/crio-f36f29cca49b4d9575a65e7103b6b62995d203eaedddd57f3760c5ae248b025e WatchSource:0}: Error finding container f36f29cca49b4d9575a65e7103b6b62995d203eaedddd57f3760c5ae248b025e: Status 404 returned error can't find the container with id f36f29cca49b4d9575a65e7103b6b62995d203eaedddd57f3760c5ae248b025e
Oct 30 22:06:47.755656 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:47.755627    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6hhw9"
Oct 30 22:06:47.755761 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:47.755655    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vs5pt"
Oct 30 22:06:47.767242 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:47.767221    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9ea63ef4-6b84-46e6-a753-bca60280734a-metrics-tls\") pod \"dns-default-c749j\" (UID: \"9ea63ef4-6b84-46e6-a753-bca60280734a\") " pod="openshift-dns/dns-default-c749j"
Oct 30 22:06:47.767354 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:47.767341    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 22:06:47.767393 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:47.767389    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ea63ef4-6b84-46e6-a753-bca60280734a-metrics-tls podName:9ea63ef4-6b84-46e6-a753-bca60280734a nodeName:}" failed. No retries permitted until 2026-10-30 22:06:48.767374392 +0000 UTC m=+32.520728630 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9ea63ef4-6b84-46e6-a753-bca60280734a-metrics-tls") pod "dns-default-c749j" (UID: "9ea63ef4-6b84-46e6-a753-bca60280734a") : secret "dns-default-metrics-tls" not found
Oct 30 22:06:47.772652 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:47.772633    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 22:06:47.772776 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:47.772667    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 22:06:47.772776 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:47.772677    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 22:06:47.772879 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:47.772788    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-c5qz2\""
Oct 30 22:06:47.772879 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:47.772811    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-ggpvh\""
Oct 30 22:06:48.004770 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:48.004721    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-wp8vp" event={"ID":"98075172-cd56-41f2-8203-fc2639e95df9","Type":"ContainerStarted","Data":"f36f29cca49b4d9575a65e7103b6b62995d203eaedddd57f3760c5ae248b025e"}
Oct 30 22:06:48.755421 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:48.755390    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rjtc8"
Oct 30 22:06:48.772134 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:48.772103    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9ea63ef4-6b84-46e6-a753-bca60280734a-metrics-tls\") pod \"dns-default-c749j\" (UID: \"9ea63ef4-6b84-46e6-a753-bca60280734a\") " pod="openshift-dns/dns-default-c749j"
Oct 30 22:06:48.772279 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:48.772261    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 22:06:48.772326 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:48.772318    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ea63ef4-6b84-46e6-a753-bca60280734a-metrics-tls podName:9ea63ef4-6b84-46e6-a753-bca60280734a nodeName:}" failed. No retries permitted until 2026-10-30 22:06:50.772304445 +0000 UTC m=+34.525658676 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9ea63ef4-6b84-46e6-a753-bca60280734a-metrics-tls") pod "dns-default-c749j" (UID: "9ea63ef4-6b84-46e6-a753-bca60280734a") : secret "dns-default-metrics-tls" not found
Oct 30 22:06:48.786588 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:48.786536    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-bf7kf\""
Oct 30 22:06:48.786713 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:48.786587    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 22:06:50.091508 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:50.091444    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-69blh"]
Oct 30 22:06:50.102980 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:50.102956    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-69blh"
Oct 30 22:06:50.119690 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:50.119661    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-69blh"]
Oct 30 22:06:50.132724 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:50.132708    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 22:06:50.132818 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:50.132762    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-26rrv\""
Oct 30 22:06:50.160342 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:50.160315    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 22:06:50.160342 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:50.160316    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 22:06:50.160458 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:50.160354    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 22:06:50.183806 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:50.183782    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b529dc48-9c57-4a0e-a4c0-49f53556ae34-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-69blh\" (UID: \"b529dc48-9c57-4a0e-a4c0-49f53556ae34\") " pod="openshift-insights/insights-runtime-extractor-69blh"
Oct 30 22:06:50.183888 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:50.183819    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2p2x4\" (UniqueName: \"kubernetes.io/projected/b529dc48-9c57-4a0e-a4c0-49f53556ae34-kube-api-access-2p2x4\") pod \"insights-runtime-extractor-69blh\" (UID: \"b529dc48-9c57-4a0e-a4c0-49f53556ae34\") " pod="openshift-insights/insights-runtime-extractor-69blh"
Oct 30 22:06:50.183888 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:50.183839    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b529dc48-9c57-4a0e-a4c0-49f53556ae34-crio-socket\") pod \"insights-runtime-extractor-69blh\" (UID: \"b529dc48-9c57-4a0e-a4c0-49f53556ae34\") " pod="openshift-insights/insights-runtime-extractor-69blh"
Oct 30 22:06:50.183953 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:50.183933    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b529dc48-9c57-4a0e-a4c0-49f53556ae34-data-volume\") pod \"insights-runtime-extractor-69blh\" (UID: \"b529dc48-9c57-4a0e-a4c0-49f53556ae34\") " pod="openshift-insights/insights-runtime-extractor-69blh"
Oct 30 22:06:50.183985 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:50.183974    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b529dc48-9c57-4a0e-a4c0-49f53556ae34-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-69blh\" (UID: \"b529dc48-9c57-4a0e-a4c0-49f53556ae34\") " pod="openshift-insights/insights-runtime-extractor-69blh"
Oct 30 22:06:50.284365 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:50.284343    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b529dc48-9c57-4a0e-a4c0-49f53556ae34-data-volume\") pod \"insights-runtime-extractor-69blh\" (UID: \"b529dc48-9c57-4a0e-a4c0-49f53556ae34\") " pod="openshift-insights/insights-runtime-extractor-69blh"
Oct 30 22:06:50.284433 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:50.284375    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b529dc48-9c57-4a0e-a4c0-49f53556ae34-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-69blh\" (UID: \"b529dc48-9c57-4a0e-a4c0-49f53556ae34\") " pod="openshift-insights/insights-runtime-extractor-69blh"
Oct 30 22:06:50.284532 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:50.284466    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 22:06:50.284532 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:50.284518    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b529dc48-9c57-4a0e-a4c0-49f53556ae34-insights-runtime-extractor-tls podName:b529dc48-9c57-4a0e-a4c0-49f53556ae34 nodeName:}" failed. No retries permitted until 2026-10-30 22:06:50.784502384 +0000 UTC m=+34.537856618 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b529dc48-9c57-4a0e-a4c0-49f53556ae34-insights-runtime-extractor-tls") pod "insights-runtime-extractor-69blh" (UID: "b529dc48-9c57-4a0e-a4c0-49f53556ae34") : secret "insights-runtime-extractor-tls" not found
Oct 30 22:06:50.284627 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:50.284541    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b529dc48-9c57-4a0e-a4c0-49f53556ae34-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-69blh\" (UID: \"b529dc48-9c57-4a0e-a4c0-49f53556ae34\") " pod="openshift-insights/insights-runtime-extractor-69blh"
Oct 30 22:06:50.284627 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:50.284575    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2p2x4\" (UniqueName: \"kubernetes.io/projected/b529dc48-9c57-4a0e-a4c0-49f53556ae34-kube-api-access-2p2x4\") pod \"insights-runtime-extractor-69blh\" (UID: \"b529dc48-9c57-4a0e-a4c0-49f53556ae34\") " pod="openshift-insights/insights-runtime-extractor-69blh"
Oct 30 22:06:50.284627 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:50.284596    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b529dc48-9c57-4a0e-a4c0-49f53556ae34-data-volume\") pod \"insights-runtime-extractor-69blh\" (UID: \"b529dc48-9c57-4a0e-a4c0-49f53556ae34\") " pod="openshift-insights/insights-runtime-extractor-69blh"
Oct 30 22:06:50.284627 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:50.284601    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b529dc48-9c57-4a0e-a4c0-49f53556ae34-crio-socket\") pod \"insights-runtime-extractor-69blh\" (UID: \"b529dc48-9c57-4a0e-a4c0-49f53556ae34\") " pod="openshift-insights/insights-runtime-extractor-69blh"
Oct 30 22:06:50.284785 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:50.284641    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 22:06:50.284785 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:50.284698    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b529dc48-9c57-4a0e-a4c0-49f53556ae34-kube-rbac-proxy-cm podName:b529dc48-9c57-4a0e-a4c0-49f53556ae34 nodeName:}" failed. No retries permitted until 2026-10-30 22:06:50.784683044 +0000 UTC m=+34.538037286 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b529dc48-9c57-4a0e-a4c0-49f53556ae34-kube-rbac-proxy-cm") pod "insights-runtime-extractor-69blh" (UID: "b529dc48-9c57-4a0e-a4c0-49f53556ae34") : configmap "kube-rbac-proxy" not found
Oct 30 22:06:50.284785 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:50.284743    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b529dc48-9c57-4a0e-a4c0-49f53556ae34-crio-socket\") pod \"insights-runtime-extractor-69blh\" (UID: \"b529dc48-9c57-4a0e-a4c0-49f53556ae34\") " pod="openshift-insights/insights-runtime-extractor-69blh"
Oct 30 22:06:50.318928 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:50.318905    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2p2x4\" (UniqueName: \"kubernetes.io/projected/b529dc48-9c57-4a0e-a4c0-49f53556ae34-kube-api-access-2p2x4\") pod \"insights-runtime-extractor-69blh\" (UID: \"b529dc48-9c57-4a0e-a4c0-49f53556ae34\") " pod="openshift-insights/insights-runtime-extractor-69blh"
Oct 30 22:06:50.352739 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:50.352715    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-nzx7v"]
Oct 30 22:06:50.361436 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:50.361417    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-nzx7v"
Oct 30 22:06:50.378770 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:50.378731    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-nzx7v"]
Oct 30 22:06:50.379116 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:50.379066    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-rrq2z\""
Oct 30 22:06:50.379369 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:50.379203    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 30 22:06:50.379425 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:50.379404    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 30 22:06:50.486736 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:50.486694    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5d6l6\" (UniqueName: \"kubernetes.io/projected/d4e7cd84-8a3f-4ef7-970c-008a9c895a47-kube-api-access-5d6l6\") pod \"migrator-66df5d496b-nzx7v\" (UID: \"d4e7cd84-8a3f-4ef7-970c-008a9c895a47\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-nzx7v"
Oct 30 22:06:50.486901 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:50.486830    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe4c45fd-690b-4111-9d15-93dc183b3e46-metrics-certs\") pod \"network-metrics-daemon-6hhw9\" (UID: \"fe4c45fd-690b-4111-9d15-93dc183b3e46\") " pod="openshift-multus/network-metrics-daemon-6hhw9"
Oct 30 22:06:50.486901 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:50.486875    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b8503b30-31d6-4454-a3fd-8bea4a527b04-original-pull-secret\") pod \"global-pull-secret-syncer-rjtc8\" (UID: \"b8503b30-31d6-4454-a3fd-8bea4a527b04\") " pod="kube-system/global-pull-secret-syncer-rjtc8"
Oct 30 22:06:50.487022 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:50.487004    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 22:06:50.487108 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:50.487097    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe4c45fd-690b-4111-9d15-93dc183b3e46-metrics-certs podName:fe4c45fd-690b-4111-9d15-93dc183b3e46 nodeName:}" failed. No retries permitted until 2026-10-30 22:07:22.487076393 +0000 UTC m=+66.240430628 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fe4c45fd-690b-4111-9d15-93dc183b3e46-metrics-certs") pod "network-metrics-daemon-6hhw9" (UID: "fe4c45fd-690b-4111-9d15-93dc183b3e46") : secret "metrics-daemon-secret" not found
Oct 30 22:06:50.489173 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:50.489155    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b8503b30-31d6-4454-a3fd-8bea4a527b04-original-pull-secret\") pod \"global-pull-secret-syncer-rjtc8\" (UID: \"b8503b30-31d6-4454-a3fd-8bea4a527b04\") " pod="kube-system/global-pull-secret-syncer-rjtc8"
Oct 30 22:06:50.570684 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:50.570652    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rjtc8"
Oct 30 22:06:50.587985 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:50.587959    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5d6l6\" (UniqueName: \"kubernetes.io/projected/d4e7cd84-8a3f-4ef7-970c-008a9c895a47-kube-api-access-5d6l6\") pod \"migrator-66df5d496b-nzx7v\" (UID: \"d4e7cd84-8a3f-4ef7-970c-008a9c895a47\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-nzx7v"
Oct 30 22:06:50.614439 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:50.614414    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5d6l6\" (UniqueName: \"kubernetes.io/projected/d4e7cd84-8a3f-4ef7-970c-008a9c895a47-kube-api-access-5d6l6\") pod \"migrator-66df5d496b-nzx7v\" (UID: \"d4e7cd84-8a3f-4ef7-970c-008a9c895a47\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-nzx7v"
Oct 30 22:06:50.687345 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:50.687315    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-nzx7v"
Oct 30 22:06:50.688574 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:50.688548    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8286n\" (UniqueName: \"kubernetes.io/projected/dbf6b14c-5576-4ae3-bbb3-b3201aa7cb7c-kube-api-access-8286n\") pod \"network-check-target-vs5pt\" (UID: \"dbf6b14c-5576-4ae3-bbb3-b3201aa7cb7c\") " pod="openshift-network-diagnostics/network-check-target-vs5pt"
Oct 30 22:06:50.690841 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:50.690821    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8286n\" (UniqueName: \"kubernetes.io/projected/dbf6b14c-5576-4ae3-bbb3-b3201aa7cb7c-kube-api-access-8286n\") pod \"network-check-target-vs5pt\" (UID: \"dbf6b14c-5576-4ae3-bbb3-b3201aa7cb7c\") " pod="openshift-network-diagnostics/network-check-target-vs5pt"
Oct 30 22:06:50.771405 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:50.771385    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vs5pt"
Oct 30 22:06:50.790870 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:50.789563    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b529dc48-9c57-4a0e-a4c0-49f53556ae34-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-69blh\" (UID: \"b529dc48-9c57-4a0e-a4c0-49f53556ae34\") " pod="openshift-insights/insights-runtime-extractor-69blh"
Oct 30 22:06:50.790870 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:50.789629    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9ea63ef4-6b84-46e6-a753-bca60280734a-metrics-tls\") pod \"dns-default-c749j\" (UID: \"9ea63ef4-6b84-46e6-a753-bca60280734a\") " pod="openshift-dns/dns-default-c749j"
Oct 30 22:06:50.790870 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:50.789659    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b529dc48-9c57-4a0e-a4c0-49f53556ae34-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-69blh\" (UID: \"b529dc48-9c57-4a0e-a4c0-49f53556ae34\") " pod="openshift-insights/insights-runtime-extractor-69blh"
Oct 30 22:06:50.790870 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:50.789741    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 22:06:50.790870 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:50.789792    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b529dc48-9c57-4a0e-a4c0-49f53556ae34-kube-rbac-proxy-cm podName:b529dc48-9c57-4a0e-a4c0-49f53556ae34 nodeName:}" failed. No retries permitted until 2026-10-30 22:06:51.789775521 +0000 UTC m=+35.543129756 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b529dc48-9c57-4a0e-a4c0-49f53556ae34-kube-rbac-proxy-cm") pod "insights-runtime-extractor-69blh" (UID: "b529dc48-9c57-4a0e-a4c0-49f53556ae34") : configmap "kube-rbac-proxy" not found
Oct 30 22:06:50.790870 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:50.790182    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 22:06:50.790870 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:50.790232    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b529dc48-9c57-4a0e-a4c0-49f53556ae34-insights-runtime-extractor-tls podName:b529dc48-9c57-4a0e-a4c0-49f53556ae34 nodeName:}" failed. No retries permitted until 2026-10-30 22:06:51.790215923 +0000 UTC m=+35.543570155 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b529dc48-9c57-4a0e-a4c0-49f53556ae34-insights-runtime-extractor-tls") pod "insights-runtime-extractor-69blh" (UID: "b529dc48-9c57-4a0e-a4c0-49f53556ae34") : secret "insights-runtime-extractor-tls" not found
Oct 30 22:06:50.790870 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:50.790279    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 22:06:50.790870 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:50.790306    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ea63ef4-6b84-46e6-a753-bca60280734a-metrics-tls podName:9ea63ef4-6b84-46e6-a753-bca60280734a nodeName:}" failed. No retries permitted until 2026-10-30 22:06:54.790297305 +0000 UTC m=+38.543651541 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9ea63ef4-6b84-46e6-a753-bca60280734a-metrics-tls") pod "dns-default-c749j" (UID: "9ea63ef4-6b84-46e6-a753-bca60280734a") : secret "dns-default-metrics-tls" not found
Oct 30 22:06:50.879056 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:50.879021    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-rjtc8"]
Oct 30 22:06:50.881917 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:50.881891    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8503b30_31d6_4454_a3fd_8bea4a527b04.slice/crio-d71ca6f17b221e257084adf9799d36cca85cac57bf35ba31239f5bd9f67fff97 WatchSource:0}: Error finding container d71ca6f17b221e257084adf9799d36cca85cac57bf35ba31239f5bd9f67fff97: Status 404 returned error can't find the container with id d71ca6f17b221e257084adf9799d36cca85cac57bf35ba31239f5bd9f67fff97
Oct 30 22:06:50.997431 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:50.997409    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-nzx7v"]
Oct 30 22:06:50.999918 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:50.999893    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4e7cd84_8a3f_4ef7_970c_008a9c895a47.slice/crio-6a85dcbaf30c440b28905b089dc95fc692b8a0ae5b68608d48f04db27b1ba940 WatchSource:0}: Error finding container 6a85dcbaf30c440b28905b089dc95fc692b8a0ae5b68608d48f04db27b1ba940: Status 404 returned error can't find the container with id 6a85dcbaf30c440b28905b089dc95fc692b8a0ae5b68608d48f04db27b1ba940
Oct 30 22:06:51.012978 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:51.012951    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-rjtc8" event={"ID":"b8503b30-31d6-4454-a3fd-8bea4a527b04","Type":"ContainerStarted","Data":"d71ca6f17b221e257084adf9799d36cca85cac57bf35ba31239f5bd9f67fff97"}
Oct 30 22:06:51.014361 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:51.014334    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-wp8vp" event={"ID":"98075172-cd56-41f2-8203-fc2639e95df9","Type":"ContainerStarted","Data":"a1276cc7b281c7dbcc12d725f606534fc61b3610330fc50bcff905cf029700ad"}
Oct 30 22:06:51.015459 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:51.015435    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-nzx7v" event={"ID":"d4e7cd84-8a3f-4ef7-970c-008a9c895a47","Type":"ContainerStarted","Data":"6a85dcbaf30c440b28905b089dc95fc692b8a0ae5b68608d48f04db27b1ba940"}
Oct 30 22:06:51.056515 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:51.056473    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-wp8vp" podStartSLOduration=1.420090467 podStartE2EDuration="4.056462536s" totalImagesPullingTime="2.636372069s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:06:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:51.055458436 +0000 UTC m=+34.808812676" watchObservedRunningTime="2026-10-30 22:06:51.056462536 +0000 UTC m=+34.809816799"
Oct 30 22:06:51.069689 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:51.069667    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-vs5pt"]
Oct 30 22:06:51.072567 ip-10-0-130-77 kubenswrapper[2579]: W1030 22:06:51.072546    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbf6b14c_5576_4ae3_bbb3_b3201aa7cb7c.slice/crio-26201244dde3aa78555ef013f144cecdb7b69913fb0d52e2e8674b4bbad190b5 WatchSource:0}: Error finding container 26201244dde3aa78555ef013f144cecdb7b69913fb0d52e2e8674b4bbad190b5: Status 404 returned error can't find the container with id 26201244dde3aa78555ef013f144cecdb7b69913fb0d52e2e8674b4bbad190b5
Oct 30 22:06:51.798892 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:51.798842    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b529dc48-9c57-4a0e-a4c0-49f53556ae34-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-69blh\" (UID: \"b529dc48-9c57-4a0e-a4c0-49f53556ae34\") " pod="openshift-insights/insights-runtime-extractor-69blh"
Oct 30 22:06:51.799655 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:51.798943    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b529dc48-9c57-4a0e-a4c0-49f53556ae34-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-69blh\" (UID: \"b529dc48-9c57-4a0e-a4c0-49f53556ae34\") " pod="openshift-insights/insights-runtime-extractor-69blh"
Oct 30 22:06:51.799655 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:51.799049    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 22:06:51.799655 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:51.799122    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b529dc48-9c57-4a0e-a4c0-49f53556ae34-kube-rbac-proxy-cm podName:b529dc48-9c57-4a0e-a4c0-49f53556ae34 nodeName:}" failed. No retries permitted until 2026-10-30 22:06:53.799101146 +0000 UTC m=+37.552455384 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b529dc48-9c57-4a0e-a4c0-49f53556ae34-kube-rbac-proxy-cm") pod "insights-runtime-extractor-69blh" (UID: "b529dc48-9c57-4a0e-a4c0-49f53556ae34") : configmap "kube-rbac-proxy" not found
Oct 30 22:06:51.799655 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:51.799192    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 22:06:51.799655 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:51.799265    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b529dc48-9c57-4a0e-a4c0-49f53556ae34-insights-runtime-extractor-tls podName:b529dc48-9c57-4a0e-a4c0-49f53556ae34 nodeName:}" failed. No retries permitted until 2026-10-30 22:06:53.799247205 +0000 UTC m=+37.552601425 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b529dc48-9c57-4a0e-a4c0-49f53556ae34-insights-runtime-extractor-tls") pod "insights-runtime-extractor-69blh" (UID: "b529dc48-9c57-4a0e-a4c0-49f53556ae34") : secret "insights-runtime-extractor-tls" not found
Oct 30 22:06:52.025275 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:52.025182    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-vs5pt" event={"ID":"dbf6b14c-5576-4ae3-bbb3-b3201aa7cb7c","Type":"ContainerStarted","Data":"26201244dde3aa78555ef013f144cecdb7b69913fb0d52e2e8674b4bbad190b5"}
Oct 30 22:06:53.036551 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:53.036481    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-nzx7v" event={"ID":"d4e7cd84-8a3f-4ef7-970c-008a9c895a47","Type":"ContainerStarted","Data":"b20e995309ec5214f76f1c9cd2787e9d93315d6cc52df03593a49bee395a51e6"}
Oct 30 22:06:53.078576 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:53.078526    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-nzx7v" podStartSLOduration=1.4155808859999999 podStartE2EDuration="3.078512479s" totalImagesPullingTime="1.662931593s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:06:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:53.077606 +0000 UTC m=+36.830960241" watchObservedRunningTime="2026-10-30 22:06:53.078512479 +0000 UTC m=+36.831866720"
Oct 30 22:06:53.816176 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:53.816138    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b529dc48-9c57-4a0e-a4c0-49f53556ae34-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-69blh\" (UID: \"b529dc48-9c57-4a0e-a4c0-49f53556ae34\") " pod="openshift-insights/insights-runtime-extractor-69blh"
Oct 30 22:06:53.816368 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:53.816205    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b529dc48-9c57-4a0e-a4c0-49f53556ae34-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-69blh\" (UID: \"b529dc48-9c57-4a0e-a4c0-49f53556ae34\") " pod="openshift-insights/insights-runtime-extractor-69blh"
Oct 30 22:06:53.816368 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:53.816279    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 22:06:53.816368 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:53.816318    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 22:06:53.816368 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:53.816357    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b529dc48-9c57-4a0e-a4c0-49f53556ae34-kube-rbac-proxy-cm podName:b529dc48-9c57-4a0e-a4c0-49f53556ae34 nodeName:}" failed. No retries permitted until 2026-10-30 22:06:57.816337019 +0000 UTC m=+41.569691259 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b529dc48-9c57-4a0e-a4c0-49f53556ae34-kube-rbac-proxy-cm") pod "insights-runtime-extractor-69blh" (UID: "b529dc48-9c57-4a0e-a4c0-49f53556ae34") : configmap "kube-rbac-proxy" not found
Oct 30 22:06:53.816586 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:53.816380    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b529dc48-9c57-4a0e-a4c0-49f53556ae34-insights-runtime-extractor-tls podName:b529dc48-9c57-4a0e-a4c0-49f53556ae34 nodeName:}" failed. No retries permitted until 2026-10-30 22:06:57.81636883 +0000 UTC m=+41.569723063 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b529dc48-9c57-4a0e-a4c0-49f53556ae34-insights-runtime-extractor-tls") pod "insights-runtime-extractor-69blh" (UID: "b529dc48-9c57-4a0e-a4c0-49f53556ae34") : secret "insights-runtime-extractor-tls" not found
Oct 30 22:06:54.823493 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:54.823426    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9ea63ef4-6b84-46e6-a753-bca60280734a-metrics-tls\") pod \"dns-default-c749j\" (UID: \"9ea63ef4-6b84-46e6-a753-bca60280734a\") " pod="openshift-dns/dns-default-c749j"
Oct 30 22:06:54.823868 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:54.823559    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 22:06:54.823868 ip-10-0-130-77 kubenswrapper[2579]: E1030 22:06:54.823617    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ea63ef4-6b84-46e6-a753-bca60280734a-metrics-tls podName:9ea63ef4-6b84-46e6-a753-bca60280734a nodeName:}" failed. No retries permitted until 2026-10-30 22:07:02.823600909 +0000 UTC m=+46.576955143 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9ea63ef4-6b84-46e6-a753-bca60280734a-metrics-tls") pod "dns-default-c749j" (UID: "9ea63ef4-6b84-46e6-a753-bca60280734a") : secret "dns-default-metrics-tls" not found
Oct 30 22:06:55.043988 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:55.043941    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-vs5pt" event={"ID":"dbf6b14c-5576-4ae3-bbb3-b3201aa7cb7c","Type":"ContainerStarted","Data":"72b59c24f3c6090d175b57af4cad78a4916d414051f5344c397815d2435b9322"}
Oct 30 22:06:55.044179 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:55.044097    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-vs5pt"
Oct 30 22:06:55.087286 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:55.087201    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-vs5pt" podStartSLOduration=35.592211366 podStartE2EDuration="39.087182658s" totalImagesPullingTime="3.494971292s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:06:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:55.08611545 +0000 UTC m=+38.839469693" watchObservedRunningTime="2026-10-30 22:06:55.087182658 +0000 UTC m=+38.840536898"
Oct 30 22:06:55.386661 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:55.386621    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mm655_a3d96078-3ec8-40aa-855a-d1ba4acf2e74/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:55.567467 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:55.567440    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mm655_a3d96078-3ec8-40aa-855a-d1ba4acf2e74/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:55.749574 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:55.749552    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mm655_a3d96078-3ec8-40aa-855a-d1ba4acf2e74/konnectivity-agent/0.log" limit=1024
Oct 30 22:06:56.019577 ip-10-0-130-77 kubenswrapper[2579]: I1030 22:06:56.019458    2579 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 22:06:56.019634 ip-10-0-130-77 systemd[1]: Stopping Kubernetes Kubelet...
Oct 30 22:06:56.032392 ip-10-0-130-77 systemd[1]: kubelet.service: Deactivated successfully.
Oct 30 22:06:56.032567 ip-10-0-130-77 systemd[1]: Stopped Kubernetes Kubelet.
Oct 30 22:06:56.032614 ip-10-0-130-77 systemd[1]: kubelet.service: Consumed 2.174s CPU time.
Oct 30 22:06:56.045510 ip-10-0-130-77 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 22:06:56.117938 ip-10-0-130-77 kubenswrapper[4477]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 22:06:56.117938 ip-10-0-130-77 kubenswrapper[4477]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 22:06:56.117938 ip-10-0-130-77 kubenswrapper[4477]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 22:06:56.117938 ip-10-0-130-77 kubenswrapper[4477]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 22:06:56.120012 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.119943    4477 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 22:06:56.120012 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.120012    4477 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 22:06:56.120098 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.120016    4477 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 22:06:56.120098 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.120019    4477 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 22:06:56.120098 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.120023    4477 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 22:06:56.120098 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.120026    4477 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 22:06:56.120098 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.120029    4477 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 22:06:56.120098 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.120031    4477 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 22:06:56.120098 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.120034    4477 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 22:06:56.120098 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.120050    4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 22:06:56.120098 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.120053    4477 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 22:06:56.120098 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.120056    4477 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 22:06:56.120098 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.120060    4477 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 22:06:56.120098 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.120062    4477 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 22:06:56.120098 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.120065    4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 22:06:56.120098 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.120067    4477 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 22:06:56.120098 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.120070    4477 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 22:06:56.120098 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.120072    4477 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 22:06:56.120098 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.120075    4477 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 22:06:56.120098 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.120078    4477 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 22:06:56.120098 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.120080    4477 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 22:06:56.120098 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.120083    4477 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 22:06:56.120585 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.120085    4477 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 22:06:56.120585 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.120088    4477 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 22:06:56.120585 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.120092    4477 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 22:06:56.120585 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.120096    4477 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 22:06:56.120585 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.120100    4477 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 22:06:56.120585 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.120108    4477 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 22:06:56.120585 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.120112    4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 22:06:56.120585 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.120114    4477 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 22:06:56.120585 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.120117    4477 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 22:06:56.120585 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.120120    4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 22:06:56.120585 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.120122    4477 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 22:06:56.120585 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.120125    4477 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 22:06:56.120585 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.120128    4477 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 22:06:56.120585 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.120130    4477 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 22:06:56.120585 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.120133    4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 22:06:56.120585 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.120135    4477 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 22:06:56.120585 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.120138    4477 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 22:06:56.120585 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.120140    4477 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 22:06:56.120585 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.120143    4477 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 22:06:56.121083 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.120145    4477 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 22:06:56.121083 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.120148    4477 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 22:06:56.121083 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.120150    4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 22:06:56.121083 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.120153    4477 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 22:06:56.121083 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.120156    4477 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 22:06:56.121083 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.120158    4477 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 22:06:56.121083 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.120166    4477 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 22:06:56.121083 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.120171    4477 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 22:06:56.121083 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.120174    4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 22:06:56.121083 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.120177    4477 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 22:06:56.121083 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.120181    4477 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 22:06:56.121083 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.120184    4477 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 22:06:56.121083 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.120187    4477 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 22:06:56.121083 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.120189    4477 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 22:06:56.121083 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.120192    4477 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 22:06:56.121083 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.120195    4477 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 22:06:56.121083 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.120198    4477 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 22:06:56.121083 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.120201    4477 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 22:06:56.121083 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.120203    4477 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 22:06:56.121539 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.120206    4477 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 22:06:56.121539 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.120209    4477 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 22:06:56.121539 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.120212    4477 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 22:06:56.121539 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.120215    4477 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 22:06:56.121539 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.120217    4477 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 22:06:56.121539 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.120220    4477 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 22:06:56.121539 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.120222    4477 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 22:06:56.121539 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.120225    4477 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 22:06:56.121539 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.120227    4477 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 22:06:56.121539 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.120230    4477 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 22:06:56.121539 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.120232    4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 22:06:56.121539 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.120235    4477 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 22:06:56.121539 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.120237    4477 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 22:06:56.121539 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.120240    4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 22:06:56.121539 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.120242    4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 22:06:56.121539 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.120245    4477 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 22:06:56.121539 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.120247    4477 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 22:06:56.121539 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.120250    4477 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 22:06:56.121539 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.120252    4477 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 22:06:56.121539 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.120255    4477 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 22:06:56.122027 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.120257    4477 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 22:06:56.122027 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.120259    4477 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 22:06:56.122027 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.120262    4477 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 22:06:56.122027 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.120265    4477 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 22:06:56.122027 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.120267    4477 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 22:06:56.122027 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.120271    4477 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 22:06:56.122027 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.120273    4477 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 22:06:56.122027 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.120276    4477 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 22:06:56.122027 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.120278    4477 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 22:06:56.122027 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.120281    4477 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 22:06:56.122027 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.120283    4477 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 22:06:56.122027 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.120286    4477 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 22:06:56.122027 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.120288    4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 22:06:56.122027 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.120291    4477 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 22:06:56.122027 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.120294    4477 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 22:06:56.122027 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.120296    4477 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 22:06:56.122027 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.120299    4477 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 22:06:56.122027 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.120301    4477 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 22:06:56.122027 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.120304    4477 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 22:06:56.122027 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.120306    4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 22:06:56.122532 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.120309    4477 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 22:06:56.122532 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.120311    4477 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 22:06:56.122532 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.120314    4477 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 22:06:56.122532 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.120316    4477 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 22:06:56.122532 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.120319    4477 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 22:06:56.122532 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.120321    4477 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 22:06:56.122532 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121036    4477 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 22:06:56.122532 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121057    4477 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 22:06:56.122532 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121061    4477 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 22:06:56.122532 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121064    4477 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 22:06:56.122532 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121066    4477 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 22:06:56.122532 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121069    4477 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 22:06:56.122532 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121071    4477 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 22:06:56.122532 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121074    4477 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 22:06:56.122532 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121077    4477 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 22:06:56.122532 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121101    4477 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 22:06:56.122532 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121105    4477 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 22:06:56.122532 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121108    4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 22:06:56.122532 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121111    4477 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 22:06:56.122532 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121114    4477 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 22:06:56.123067 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121116    4477 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 22:06:56.123067 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121119    4477 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 22:06:56.123067 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121122    4477 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 22:06:56.123067 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121124    4477 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 22:06:56.123067 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121127    4477 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 22:06:56.123067 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121129    4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 22:06:56.123067 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121132    4477 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 22:06:56.123067 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121135    4477 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 22:06:56.123067 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121138    4477 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 22:06:56.123067 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121141    4477 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 22:06:56.123067 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121143    4477 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 22:06:56.123067 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121146    4477 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 22:06:56.123067 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121150    4477 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 22:06:56.123067 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121153    4477 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 22:06:56.123067 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121156    4477 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 22:06:56.123067 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121158    4477 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 22:06:56.123067 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121161    4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 22:06:56.123067 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121164    4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 22:06:56.123067 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121166    4477 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 22:06:56.123536 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121169    4477 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 22:06:56.123536 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121171    4477 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 22:06:56.123536 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121175    4477 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 22:06:56.123536 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121178    4477 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 22:06:56.123536 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121180    4477 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 22:06:56.123536 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121183    4477 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 22:06:56.123536 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121186    4477 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 22:06:56.123536 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121189    4477 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 22:06:56.123536 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121195    4477 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 22:06:56.123536 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121198    4477 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 22:06:56.123536 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121201    4477 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 22:06:56.123536 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121203    4477 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 22:06:56.123536 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121206    4477 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 22:06:56.123536 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121209    4477 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 22:06:56.123536 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121211    4477 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 22:06:56.123536 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121214    4477 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 22:06:56.123536 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121216    4477 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 22:06:56.123536 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121219    4477 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 22:06:56.123536 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121221    4477 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 22:06:56.123536 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121224    4477 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 22:06:56.124029 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121227    4477 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 22:06:56.124029 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121230    4477 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 22:06:56.124029 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121232    4477 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 22:06:56.124029 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121235    4477 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 22:06:56.124029 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121237    4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 22:06:56.124029 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121239    4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 22:06:56.124029 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121242    4477 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 22:06:56.124029 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121245    4477 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 22:06:56.124029 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121247    4477 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 22:06:56.124029 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121250    4477 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 22:06:56.124029 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121252    4477 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 22:06:56.124029 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121255    4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 22:06:56.124029 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121257    4477 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 22:06:56.124029 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121260    4477 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 22:06:56.124029 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121262    4477 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 22:06:56.124029 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121265    4477 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 22:06:56.124029 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121267    4477 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 22:06:56.124029 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121270    4477 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 22:06:56.124029 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121273    4477 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 22:06:56.124029 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121276    4477 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 22:06:56.124541 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121280    4477 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 22:06:56.124541 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121283    4477 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 22:06:56.124541 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121286    4477 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 22:06:56.124541 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121289    4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 22:06:56.124541 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121291    4477 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 22:06:56.124541 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121294    4477 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 22:06:56.124541 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121296    4477 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 22:06:56.124541 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121299    4477 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 22:06:56.124541 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121301    4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 22:06:56.124541 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121304    4477 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 22:06:56.124541 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121307    4477 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 22:06:56.124541 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121310    4477 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 22:06:56.124541 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121313    4477 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 22:06:56.124541 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121316    4477 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 22:06:56.124541 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121318    4477 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 22:06:56.124541 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121321    4477 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 22:06:56.124541 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121323    4477 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 22:06:56.124541 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121326    4477 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 22:06:56.124541 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121328    4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 22:06:56.124541 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121331    4477 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 22:06:56.125053 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121333    4477 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 22:06:56.125053 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121336    4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 22:06:56.125053 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121338    4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 22:06:56.125053 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121341    4477 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 22:06:56.125053 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121343    4477 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 22:06:56.125053 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121348    4477 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 22:06:56.125053 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121352    4477 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 22:06:56.125053 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121355    4477 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 22:06:56.125053 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121358    4477 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 22:06:56.125053 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121361    4477 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 22:06:56.125053 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121364    4477 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 22:06:56.125053 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121368    4477 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 22:06:56.125053 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121372    4477 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 22:06:56.125053 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121846    4477 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 22:06:56.125053 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121852    4477 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 22:06:56.125053 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121855    4477 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 22:06:56.125053 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121857    4477 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 22:06:56.125053 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121860    4477 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 22:06:56.125053 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121863    4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 22:06:56.125639 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121865    4477 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 22:06:56.125639 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121868    4477 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 22:06:56.125639 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121870    4477 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 22:06:56.125639 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121873    4477 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 22:06:56.125639 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121875    4477 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 22:06:56.125639 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121878    4477 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 22:06:56.125639 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121881    4477 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 22:06:56.125639 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121884    4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 22:06:56.125639 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121886    4477 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 22:06:56.125639 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121889    4477 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 22:06:56.125639 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121891    4477 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 22:06:56.125639 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121894    4477 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 22:06:56.125639 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121897    4477 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 22:06:56.125639 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121899    4477 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 22:06:56.125639 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121902    4477 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 22:06:56.125639 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121904    4477 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 22:06:56.125639 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121908    4477 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 22:06:56.125639 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121912    4477 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 22:06:56.125639 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121916    4477 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 22:06:56.126493 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121919    4477 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 22:06:56.126493 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121922    4477 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 22:06:56.126493 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121925    4477 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 22:06:56.126493 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121927    4477 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 22:06:56.126493 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121930    4477 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 22:06:56.126493 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121932    4477 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 22:06:56.126493 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121937    4477 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 22:06:56.126493 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121940    4477 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 22:06:56.126493 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121942    4477 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 22:06:56.126493 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121945    4477 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 22:06:56.126493 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121947    4477 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 22:06:56.126493 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121950    4477 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 22:06:56.126493 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121952    4477 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 22:06:56.126493 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121955    4477 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 22:06:56.126493 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121958    4477 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 22:06:56.126493 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121960    4477 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 22:06:56.126493 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121963    4477 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 22:06:56.126493 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121965    4477 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 22:06:56.126493 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121968    4477 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 22:06:56.126493 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121970    4477 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 22:06:56.127353 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121973    4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 22:06:56.127353 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121976    4477 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 22:06:56.127353 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121978    4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 22:06:56.127353 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121981    4477 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 22:06:56.127353 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121984    4477 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 22:06:56.127353 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121986    4477 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 22:06:56.127353 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121990    4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 22:06:56.127353 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121993    4477 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 22:06:56.127353 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121995    4477 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 22:06:56.127353 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.121998    4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 22:06:56.127353 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.122001    4477 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 22:06:56.127353 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.122003    4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 22:06:56.127353 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.122005    4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 22:06:56.127353 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.122008    4477 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 22:06:56.127353 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.122010    4477 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 22:06:56.127353 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.122013    4477 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 22:06:56.127353 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.122015    4477 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 22:06:56.127353 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.122018    4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 22:06:56.127353 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.122021    4477 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 22:06:56.128198 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.122024    4477 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 22:06:56.128198 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.122027    4477 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 22:06:56.128198 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.122029    4477 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 22:06:56.128198 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.122032    4477 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 22:06:56.128198 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.122035    4477 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 22:06:56.128198 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.122050    4477 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 22:06:56.128198 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.122053    4477 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 22:06:56.128198 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.122056    4477 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 22:06:56.128198 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.122058    4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 22:06:56.128198 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.122061    4477 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 22:06:56.128198 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.122064    4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 22:06:56.128198 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.122066    4477 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 22:06:56.128198 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.122070    4477 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 22:06:56.128198 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.122072    4477 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 22:06:56.128198 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.122075    4477 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 22:06:56.128198 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.122078    4477 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 22:06:56.128198 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.122081    4477 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 22:06:56.128198 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.122083    4477 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 22:06:56.128198 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.122086    4477 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 22:06:56.128198 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.122090    4477 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 22:06:56.128949 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.122092    4477 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 22:06:56.128949 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.122095    4477 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 22:06:56.128949 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.122097    4477 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 22:06:56.128949 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.122100    4477 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 22:06:56.128949 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.122102    4477 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 22:06:56.128949 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.122105    4477 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 22:06:56.128949 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.122107    4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 22:06:56.128949 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.122110    4477 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 22:06:56.128949 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.122112    4477 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 22:06:56.128949 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.122115    4477 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 22:06:56.128949 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.122117    4477 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 22:06:56.128949 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.122122    4477 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 22:06:56.128949 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.122124    4477 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 22:06:56.128949 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.122127    4477 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 22:06:56.128949 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.122130    4477 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 22:06:56.128949 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.122132    4477 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 22:06:56.128949 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.122135    4477 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 22:06:56.128949 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.122137    4477 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 22:06:56.128949 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.122140    4477 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 22:06:56.129442 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.122142    4477 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 22:06:56.129442 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.122145    4477 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 22:06:56.129442 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.122148    4477 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 22:06:56.129442 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122251    4477 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 22:06:56.129442 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122270    4477 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 22:06:56.129442 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122279    4477 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 22:06:56.129442 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122284    4477 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 22:06:56.129442 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122289    4477 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 22:06:56.129442 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122293    4477 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 22:06:56.129442 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122298    4477 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 22:06:56.129442 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122303    4477 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 22:06:56.129442 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122306    4477 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 22:06:56.129442 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122315    4477 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 22:06:56.129442 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122319    4477 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 22:06:56.129442 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122322    4477 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 22:06:56.129442 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122326    4477 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 22:06:56.129442 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122329    4477 flags.go:64] FLAG: --cgroup-root=""
Oct 30 22:06:56.129442 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122332    4477 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 22:06:56.129442 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122335    4477 flags.go:64] FLAG: --client-ca-file=""
Oct 30 22:06:56.129442 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122338    4477 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 22:06:56.129442 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122341    4477 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 22:06:56.129442 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122345    4477 flags.go:64] FLAG: --cluster-domain=""
Oct 30 22:06:56.129442 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122348    4477 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 22:06:56.129442 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122351    4477 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 22:06:56.130079 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122355    4477 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 22:06:56.130079 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122359    4477 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 22:06:56.130079 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122363    4477 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 22:06:56.130079 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122366    4477 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 22:06:56.130079 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122369    4477 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 22:06:56.130079 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122372    4477 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 22:06:56.130079 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122375    4477 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 22:06:56.130079 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122378    4477 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 22:06:56.130079 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122381    4477 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 22:06:56.130079 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122384    4477 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 22:06:56.130079 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122387    4477 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 22:06:56.130079 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122392    4477 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 22:06:56.130079 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122395    4477 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 22:06:56.130079 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122397    4477 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 22:06:56.130079 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122400    4477 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 22:06:56.130079 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122403    4477 flags.go:64] FLAG: --enable-server="true"
Oct 30 22:06:56.130079 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122406    4477 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 22:06:56.130079 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122411    4477 flags.go:64] FLAG: --event-burst="100"
Oct 30 22:06:56.130079 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122415    4477 flags.go:64] FLAG: --event-qps="50"
Oct 30 22:06:56.130079 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122417    4477 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 22:06:56.130079 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122422    4477 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 22:06:56.130079 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122425    4477 flags.go:64] FLAG: --eviction-hard=""
Oct 30 22:06:56.130079 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122429    4477 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 22:06:56.130079 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122432    4477 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 22:06:56.130668 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122435    4477 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 22:06:56.130668 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122438    4477 flags.go:64] FLAG: --eviction-soft=""
Oct 30 22:06:56.130668 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122441    4477 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 22:06:56.130668 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122444    4477 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 22:06:56.130668 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122447    4477 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 22:06:56.130668 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122450    4477 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 22:06:56.130668 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122453    4477 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 22:06:56.130668 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122456    4477 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 22:06:56.130668 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122461    4477 flags.go:64] FLAG: --feature-gates=""
Oct 30 22:06:56.130668 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122465    4477 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 22:06:56.130668 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122468    4477 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 22:06:56.130668 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122472    4477 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 22:06:56.130668 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122475    4477 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 22:06:56.130668 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122478    4477 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 22:06:56.130668 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122481    4477 flags.go:64] FLAG: --help="false"
Oct 30 22:06:56.130668 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122484    4477 flags.go:64] FLAG: --hostname-override="ip-10-0-130-77.ec2.internal"
Oct 30 22:06:56.130668 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122487    4477 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 22:06:56.130668 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122490    4477 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 22:06:56.130668 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122493    4477 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 22:06:56.130668 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122497    4477 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 22:06:56.130668 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122500    4477 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 22:06:56.130668 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122503    4477 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 22:06:56.130668 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122506    4477 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 22:06:56.131355 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122509    4477 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 22:06:56.131355 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122512    4477 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 22:06:56.131355 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122515    4477 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 22:06:56.131355 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122519    4477 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 22:06:56.131355 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122522    4477 flags.go:64] FLAG: --kube-reserved=""
Oct 30 22:06:56.131355 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122527    4477 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 22:06:56.131355 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122529    4477 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 22:06:56.131355 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122533    4477 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 22:06:56.131355 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122536    4477 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 22:06:56.131355 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122539    4477 flags.go:64] FLAG: --lock-file=""
Oct 30 22:06:56.131355 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122542    4477 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 22:06:56.131355 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122545    4477 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 22:06:56.131355 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122548    4477 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 22:06:56.131355 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122553    4477 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 22:06:56.131355 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122557    4477 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 22:06:56.131355 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122560    4477 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 22:06:56.131355 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122563    4477 flags.go:64] FLAG: --logging-format="text"
Oct 30 22:06:56.131355 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122566    4477 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 22:06:56.131355 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122570    4477 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 22:06:56.131355 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122572    4477 flags.go:64] FLAG: --manifest-url=""
Oct 30 22:06:56.131355 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122575    4477 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 22:06:56.131355 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122580    4477 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 22:06:56.131355 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122583    4477 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 22:06:56.131355 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122587    4477 flags.go:64] FLAG: --max-pods="110"
Oct 30 22:06:56.131355 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122590    4477 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 22:06:56.131969 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122593    4477 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 22:06:56.131969 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122596    4477 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 22:06:56.131969 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122598    4477 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 22:06:56.131969 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122601    4477 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 22:06:56.131969 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122604    4477 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 22:06:56.131969 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122607    4477 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 22:06:56.131969 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122614    4477 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 22:06:56.131969 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122617    4477 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 22:06:56.131969 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122620    4477 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 22:06:56.131969 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122623    4477 flags.go:64] FLAG: --pod-cidr=""
Oct 30 22:06:56.131969 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122626    4477 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 22:06:56.131969 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122629    4477 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 22:06:56.131969 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122633    4477 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 22:06:56.131969 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122636    4477 flags.go:64] FLAG: --port="10250"
Oct 30 22:06:56.131969 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122639    4477 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 22:06:56.131969 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122642    4477 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-00158a26bf59842fb"
Oct 30 22:06:56.131969 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122645    4477 flags.go:64] FLAG: --qos-reserved=""
Oct 30 22:06:56.131969 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122648    4477 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 22:06:56.131969 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122651    4477 flags.go:64] FLAG: --register-node="true"
Oct 30 22:06:56.131969 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122654    4477 flags.go:64] FLAG: --register-with-taints=""
Oct 30 22:06:56.131969 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122657    4477 flags.go:64] FLAG: --registry-burst="10"
Oct 30 22:06:56.131969 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122660    4477 flags.go:64] FLAG: --registry-qps="5"
Oct 30 22:06:56.131969 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122664    4477 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 22:06:56.131969 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122667    4477 flags.go:64] FLAG: --reserved-memory=""
Oct 30 22:06:56.131969 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122671    4477 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 22:06:56.131969 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122674    4477 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 22:06:56.132724 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122677    4477 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 22:06:56.132724 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122680    4477 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 22:06:56.132724 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122683    4477 flags.go:64] FLAG: --runonce="false"
Oct 30 22:06:56.132724 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122686    4477 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 22:06:56.132724 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122689    4477 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 22:06:56.132724 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122692    4477 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 22:06:56.132724 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122695    4477 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 22:06:56.132724 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122698    4477 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 22:06:56.132724 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122701    4477 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 22:06:56.132724 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122704    4477 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 22:06:56.132724 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122707    4477 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 22:06:56.132724 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122710    4477 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 22:06:56.132724 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122713    4477 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 22:06:56.132724 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122716    4477 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 22:06:56.132724 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122719    4477 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 22:06:56.132724 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122722    4477 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 22:06:56.132724 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122725    4477 flags.go:64] FLAG: --system-cgroups=""
Oct 30 22:06:56.132724 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122728    4477 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 22:06:56.132724 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122733    4477 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 22:06:56.132724 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122737    4477 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 22:06:56.132724 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122740    4477 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 22:06:56.132724 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122744    4477 flags.go:64] FLAG: --tls-min-version=""
Oct 30 22:06:56.132724 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122747    4477 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 22:06:56.132724 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122750    4477 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 22:06:56.132724 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122752    4477 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 22:06:56.133342 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122755    4477 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 22:06:56.133342 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122758    4477 flags.go:64] FLAG: --v="2"
Oct 30 22:06:56.133342 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122763    4477 flags.go:64] FLAG: --version="false"
Oct 30 22:06:56.133342 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122766    4477 flags.go:64] FLAG: --vmodule=""
Oct 30 22:06:56.133342 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122770    4477 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 22:06:56.133342 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.122774    4477 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 22:06:56.133342 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.122906    4477 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 22:06:56.133342 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.122909    4477 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 22:06:56.133342 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.122912    4477 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 22:06:56.133342 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.122915    4477 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 22:06:56.133342 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.122918    4477 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 22:06:56.133342 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.122920    4477 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 22:06:56.133342 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.122923    4477 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 22:06:56.133342 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.122925    4477 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 22:06:56.133342 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.122927    4477 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 22:06:56.133342 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.122931    4477 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 22:06:56.133342 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.122935    4477 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 22:06:56.133342 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.122938    4477 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 22:06:56.133342 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.122941    4477 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 22:06:56.133342 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.122944    4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 22:06:56.133874 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.122946    4477 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 22:06:56.133874 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.122948    4477 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 22:06:56.133874 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.122951    4477 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 22:06:56.133874 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.122953    4477 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 22:06:56.133874 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.122956    4477 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 22:06:56.133874 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.122959    4477 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 22:06:56.133874 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.122962    4477 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 22:06:56.133874 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.122964    4477 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 22:06:56.133874 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.122967    4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 22:06:56.133874 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.122969    4477 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 22:06:56.133874 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.122972    4477 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 22:06:56.133874 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.122975    4477 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 22:06:56.133874 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.122978    4477 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 22:06:56.133874 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.122980    4477 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 22:06:56.133874 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.122983    4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 22:06:56.133874 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.122985    4477 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 22:06:56.133874 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.122988    4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 22:06:56.133874 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.122990    4477 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 22:06:56.133874 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.122993    4477 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 22:06:56.133874 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.122995    4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 22:06:56.134412 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.122998    4477 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 22:06:56.134412 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123000    4477 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 22:06:56.134412 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123003    4477 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 22:06:56.134412 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123005    4477 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 22:06:56.134412 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123008    4477 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 22:06:56.134412 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123011    4477 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 22:06:56.134412 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123013    4477 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 22:06:56.134412 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123016    4477 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 22:06:56.134412 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123018    4477 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 22:06:56.134412 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123021    4477 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 22:06:56.134412 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123023    4477 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 22:06:56.134412 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123027    4477 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 22:06:56.134412 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123031    4477 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 22:06:56.134412 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123034    4477 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 22:06:56.134412 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123049    4477 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 22:06:56.134412 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123052    4477 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 22:06:56.134412 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123055    4477 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 22:06:56.134412 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123057    4477 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 22:06:56.134412 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123060    4477 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 22:06:56.134912 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123063    4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 22:06:56.134912 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123068    4477 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 22:06:56.134912 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123071    4477 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 22:06:56.134912 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123074    4477 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 22:06:56.134912 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123077    4477 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 22:06:56.134912 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123080    4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 22:06:56.134912 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123082    4477 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 22:06:56.134912 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123085    4477 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 22:06:56.134912 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123088    4477 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 22:06:56.134912 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123090    4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 22:06:56.134912 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123093    4477 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 22:06:56.134912 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123096    4477 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 22:06:56.134912 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123098    4477 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 22:06:56.134912 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123101    4477 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 22:06:56.134912 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123103    4477 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 22:06:56.134912 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123106    4477 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 22:06:56.134912 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123109    4477 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 22:06:56.134912 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123111    4477 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 22:06:56.134912 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123114    4477 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 22:06:56.134912 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123116    4477 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 22:06:56.135405 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123119    4477 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 22:06:56.135405 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123122    4477 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 22:06:56.135405 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123124    4477 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 22:06:56.135405 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123126    4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 22:06:56.135405 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123130    4477 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 22:06:56.135405 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123133    4477 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 22:06:56.135405 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123135    4477 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 22:06:56.135405 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123138    4477 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 22:06:56.135405 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123140    4477 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 22:06:56.135405 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123143    4477 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 22:06:56.135405 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123145    4477 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 22:06:56.135405 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123148    4477 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 22:06:56.135405 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123151    4477 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 22:06:56.135405 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123155    4477 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 22:06:56.135405 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123158    4477 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 22:06:56.135405 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123161    4477 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 22:06:56.135405 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123163    4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 22:06:56.135405 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123166    4477 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 22:06:56.135405 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123169    4477 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 22:06:56.135405 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123171    4477 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 22:06:56.135990 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123174    4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 22:06:56.135990 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123177    4477 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 22:06:56.135990 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123179    4477 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 22:06:56.135990 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123182    4477 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 22:06:56.135990 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123184    4477 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 22:06:56.135990 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123187    4477 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 22:06:56.135990 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123190    4477 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 22:06:56.135990 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123193    4477 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 22:06:56.135990 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123195    4477 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 22:06:56.135990 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123198    4477 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 22:06:56.135990 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123200    4477 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 22:06:56.135990 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123203    4477 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 22:06:56.135990 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123206    4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 22:06:56.135990 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.123231    4477 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 22:06:56.135990 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123264    4477 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 22:06:56.135990 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123268    4477 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 22:06:56.136489 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123274    4477 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 22:06:56.136489 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123277    4477 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 22:06:56.136489 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123279    4477 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 22:06:56.136489 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123283    4477 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 22:06:56.136489 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123287    4477 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 22:06:56.136489 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123291    4477 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 22:06:56.136489 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123294    4477 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 22:06:56.136489 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123297    4477 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 22:06:56.136489 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123299    4477 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 22:06:56.136489 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123302    4477 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 22:06:56.136489 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123306    4477 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 22:06:56.136489 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123309    4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 22:06:56.136489 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123312    4477 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 22:06:56.136489 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123314    4477 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 22:06:56.136489 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123317    4477 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 22:06:56.136489 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123320    4477 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 22:06:56.136489 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123322    4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 22:06:56.136489 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123325    4477 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 22:06:56.136489 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123327    4477 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 22:06:56.137025 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123330    4477 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 22:06:56.137025 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123332    4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 22:06:56.137025 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123335    4477 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 22:06:56.137025 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123338    4477 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 22:06:56.137025 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123340    4477 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 22:06:56.137025 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123343    4477 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 22:06:56.137025 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123346    4477 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 22:06:56.137025 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123348    4477 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 22:06:56.137025 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123351    4477 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 22:06:56.137025 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123354    4477 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 22:06:56.137025 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123356    4477 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 22:06:56.137025 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123359    4477 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 22:06:56.137025 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123361    4477 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 22:06:56.137025 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123365    4477 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 22:06:56.137025 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123368    4477 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 22:06:56.137025 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123371    4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 22:06:56.137025 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123373    4477 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 22:06:56.137025 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123376    4477 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 22:06:56.137025 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123379    4477 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 22:06:56.137025 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123381    4477 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 22:06:56.137561 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123384    4477 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 22:06:56.137561 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123386    4477 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 22:06:56.137561 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123389    4477 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 22:06:56.137561 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123393    4477 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 22:06:56.137561 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123396    4477 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 22:06:56.137561 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123399    4477 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 22:06:56.137561 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123401    4477 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 22:06:56.137561 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123404    4477 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 22:06:56.137561 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123407    4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 22:06:56.137561 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123409    4477 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 22:06:56.137561 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123412    4477 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 22:06:56.137561 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123414    4477 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 22:06:56.137561 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123417    4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 22:06:56.137561 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123419    4477 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 22:06:56.137561 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123422    4477 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 22:06:56.137561 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123425    4477 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 22:06:56.137561 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123428    4477 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 22:06:56.137561 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123431    4477 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 22:06:56.137561 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123433    4477 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 22:06:56.137561 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123436    4477 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 22:06:56.138120 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123439    4477 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 22:06:56.138120 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123441    4477 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 22:06:56.138120 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123444    4477 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 22:06:56.138120 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123446    4477 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 22:06:56.138120 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123449    4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 22:06:56.138120 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123453    4477 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 22:06:56.138120 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123456    4477 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 22:06:56.138120 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123458    4477 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 22:06:56.138120 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123461    4477 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 22:06:56.138120 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123463    4477 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 22:06:56.138120 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123466    4477 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 22:06:56.138120 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123468    4477 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 22:06:56.138120 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123471    4477 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 22:06:56.138120 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123473    4477 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 22:06:56.138120 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123477    4477 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 22:06:56.138120 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123482    4477 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 22:06:56.138120 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123485    4477 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 22:06:56.138120 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123488    4477 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 22:06:56.138120 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123491    4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 22:06:56.138587 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123493    4477 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 22:06:56.138587 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123496    4477 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 22:06:56.138587 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123499    4477 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 22:06:56.138587 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123501    4477 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 22:06:56.138587 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123504    4477 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 22:06:56.138587 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123506    4477 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 22:06:56.138587 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123509    4477 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 22:06:56.138587 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123511    4477 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 22:06:56.138587 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123514    4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 22:06:56.138587 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123517    4477 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 22:06:56.138587 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123519    4477 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 22:06:56.138587 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123522    4477 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 22:06:56.138587 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123525    4477 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 22:06:56.138587 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123527    4477 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 22:06:56.138587 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123530    4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 22:06:56.138587 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123532    4477 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 22:06:56.138587 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123534    4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 22:06:56.138587 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123537    4477 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 22:06:56.138587 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123542    4477 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 22:06:56.138587 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123544    4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 22:06:56.139075 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123547    4477 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 22:06:56.139075 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123550    4477 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 22:06:56.139075 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123553    4477 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 22:06:56.139075 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123555    4477 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 22:06:56.139075 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123558    4477 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 22:06:56.139075 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.123561    4477 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 22:06:56.139075 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.128015    4477 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 22:06:56.139075 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.128035    4477 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 22:06:56.139075 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.128075    4477 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 22:06:56.139075 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.128082    4477 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 22:06:56.139075 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.128338    4477 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 22:06:56.139075 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.128348    4477 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 22:06:56.139075 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.128355    4477 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 22:06:56.139075 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.128360    4477 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 22:06:56.139075 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.128365    4477 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 22:06:56.139075 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.128370    4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 22:06:56.139075 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.128381    4477 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 22:06:56.139075 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.128386    4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 22:06:56.139075 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.128420    4477 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 22:06:56.139539 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.128426    4477 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 22:06:56.139539 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.128431    4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 22:06:56.139539 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.128436    4477 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 22:06:56.139539 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.128440    4477 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 22:06:56.139539 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.128445    4477 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 22:06:56.139539 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.128449    4477 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 22:06:56.139539 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.128454    4477 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 22:06:56.139539 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.128459    4477 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 22:06:56.139539 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.128463    4477 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 22:06:56.139539 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.128468    4477 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 22:06:56.139539 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.128479    4477 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 22:06:56.139539 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.128483    4477 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 22:06:56.139539 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.128488    4477 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 22:06:56.139539 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.128493    4477 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 22:06:56.139539 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.128498    4477 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 22:06:56.139539 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.128502    4477 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 22:06:56.139539 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.128507    4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 22:06:56.139539 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.128511    4477 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 22:06:56.139539 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.128516    4477 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 22:06:56.139539 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.128521    4477 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 22:06:56.140086 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.128525    4477 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 22:06:56.140086 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.128530    4477 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 22:06:56.140086 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.128540    4477 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 22:06:56.140086 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.128544    4477 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 22:06:56.140086 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.128549    4477 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 22:06:56.140086 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.128553    4477 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 22:06:56.140086 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.128558    4477 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 22:06:56.140086 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.128563    4477 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 22:06:56.140086 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.128568    4477 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 22:06:56.140086 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.128572    4477 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 22:06:56.140086 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.128579    4477 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 22:06:56.140086 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.128587    4477 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 22:06:56.140086 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.128628    4477 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 22:06:56.140086 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.128635    4477 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 22:06:56.140086 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.128641    4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 22:06:56.140086 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.128652    4477 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 22:06:56.140086 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.128656    4477 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 22:06:56.140086 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.128661    4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 22:06:56.140086 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.128666    4477 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 22:06:56.140701 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.128671    4477 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 22:06:56.140701 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.128675    4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 22:06:56.140701 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.128682    4477 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 22:06:56.140701 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.128687    4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 22:06:56.140701 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.128692    4477 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 22:06:56.140701 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.128697    4477 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 22:06:56.140701 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.128701    4477 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 22:06:56.140701 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.128706    4477 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 22:06:56.140701 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.128764    4477 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 22:06:56.140701 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.128906    4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 22:06:56.140701 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.128917    4477 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 22:06:56.140701 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.128923    4477 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 22:06:56.140701 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.128927    4477 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 22:06:56.140701 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.128932    4477 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 22:06:56.140701 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.128936    4477 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 22:06:56.140701 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.128943    4477 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 22:06:56.140701 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.128948    4477 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 22:06:56.140701 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.128952    4477 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 22:06:56.140701 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.128956    4477 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 22:06:56.141312 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.128961    4477 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 22:06:56.141312 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.128965    4477 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 22:06:56.141312 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.128969    4477 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 22:06:56.141312 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.128974    4477 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 22:06:56.141312 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.128978    4477 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 22:06:56.141312 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.128982    4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 22:06:56.141312 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.128986    4477 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 22:06:56.141312 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.128991    4477 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 22:06:56.141312 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.128996    4477 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 22:06:56.141312 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129001    4477 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 22:06:56.141312 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129005    4477 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 22:06:56.141312 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129009    4477 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 22:06:56.141312 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129016    4477 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 22:06:56.141312 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129020    4477 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 22:06:56.141312 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129024    4477 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 22:06:56.141312 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129028    4477 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 22:06:56.141312 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129032    4477 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 22:06:56.141312 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129037    4477 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 22:06:56.141312 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129055    4477 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 22:06:56.141312 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129059    4477 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 22:06:56.141977 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129063    4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 22:06:56.141977 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129068    4477 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 22:06:56.141977 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129077    4477 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 22:06:56.141977 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129082    4477 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 22:06:56.141977 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129086    4477 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 22:06:56.141977 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129090    4477 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 22:06:56.141977 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129095    4477 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 22:06:56.141977 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129099    4477 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 22:06:56.141977 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129103    4477 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 22:06:56.141977 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129108    4477 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 22:06:56.141977 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129112    4477 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 22:06:56.141977 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129116    4477 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 22:06:56.141977 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129120    4477 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 22:06:56.141977 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129125    4477 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 22:06:56.141977 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129130    4477 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 22:06:56.141977 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129134    4477 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 22:06:56.141977 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129138    4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 22:06:56.141977 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129142    4477 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 22:06:56.141977 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129147    4477 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 22:06:56.142653 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.129185    4477 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 22:06:56.142653 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129334    4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 22:06:56.142653 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129338    4477 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 22:06:56.142653 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129342    4477 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 22:06:56.142653 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129345    4477 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 22:06:56.142653 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129348    4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 22:06:56.142653 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129351    4477 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 22:06:56.142653 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129354    4477 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 22:06:56.142653 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129356    4477 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 22:06:56.142653 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129359    4477 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 22:06:56.142653 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129362    4477 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 22:06:56.142653 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129365    4477 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 22:06:56.142653 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129367    4477 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 22:06:56.142653 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129370    4477 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 22:06:56.142653 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129373    4477 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 22:06:56.142653 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129376    4477 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 22:06:56.143175 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129380    4477 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 22:06:56.143175 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129383    4477 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 22:06:56.143175 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129385    4477 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 22:06:56.143175 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129388    4477 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 22:06:56.143175 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129391    4477 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 22:06:56.143175 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129393    4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 22:06:56.143175 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129396    4477 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 22:06:56.143175 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129399    4477 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 22:06:56.143175 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129402    4477 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 22:06:56.143175 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129404    4477 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 22:06:56.143175 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129407    4477 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 22:06:56.143175 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129410    4477 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 22:06:56.143175 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129412    4477 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 22:06:56.143175 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129415    4477 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 22:06:56.143175 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129418    4477 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 22:06:56.143175 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129421    4477 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 22:06:56.143175 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129423    4477 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 22:06:56.143175 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129426    4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 22:06:56.143175 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129429    4477 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 22:06:56.143640 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129431    4477 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 22:06:56.143640 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129434    4477 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 22:06:56.143640 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129437    4477 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 22:06:56.143640 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129439    4477 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 22:06:56.143640 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129443    4477 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 22:06:56.143640 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129446    4477 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 22:06:56.143640 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129449    4477 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 22:06:56.143640 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129453    4477 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 22:06:56.143640 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129456    4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 22:06:56.143640 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129460    4477 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 22:06:56.143640 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129464    4477 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 22:06:56.143640 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129467    4477 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 22:06:56.143640 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129470    4477 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 22:06:56.143640 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129474    4477 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 22:06:56.143640 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129477    4477 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 22:06:56.143640 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129480    4477 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 22:06:56.143640 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129482    4477 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 22:06:56.143640 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129485    4477 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 22:06:56.143640 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129487    4477 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 22:06:56.144220 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129490    4477 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 22:06:56.144220 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129493    4477 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 22:06:56.144220 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129495    4477 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 22:06:56.144220 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129498    4477 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 22:06:56.144220 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129501    4477 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 22:06:56.144220 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129505    4477 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 22:06:56.144220 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129509    4477 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 22:06:56.144220 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129512    4477 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 22:06:56.144220 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129515    4477 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 22:06:56.144220 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129518    4477 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 22:06:56.144220 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129521    4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 22:06:56.144220 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129523    4477 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 22:06:56.144220 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129526    4477 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 22:06:56.144220 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129528    4477 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 22:06:56.144220 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129531    4477 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 22:06:56.144220 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129534    4477 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 22:06:56.144220 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129537    4477 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 22:06:56.144220 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129539    4477 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 22:06:56.144220 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129542    4477 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 22:06:56.144220 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129545    4477 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 22:06:56.145010 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129548    4477 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 22:06:56.145010 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129550    4477 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 22:06:56.145010 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129553    4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 22:06:56.145010 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129556    4477 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 22:06:56.145010 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129558    4477 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 22:06:56.145010 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129561    4477 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 22:06:56.145010 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129564    4477 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 22:06:56.145010 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129567    4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 22:06:56.145010 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129570    4477 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 22:06:56.145010 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129573    4477 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 22:06:56.145010 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129575    4477 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 22:06:56.145010 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129578    4477 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 22:06:56.145010 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129580    4477 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 22:06:56.145010 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129583    4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 22:06:56.145010 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129585    4477 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 22:06:56.145010 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129588    4477 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 22:06:56.145010 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129591    4477 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 22:06:56.145010 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129593    4477 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 22:06:56.145010 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129596    4477 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 22:06:56.145010 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129599    4477 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 22:06:56.145553 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129601    4477 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 22:06:56.145553 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129604    4477 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 22:06:56.145553 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129606    4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 22:06:56.145553 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129609    4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 22:06:56.145553 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129612    4477 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 22:06:56.145553 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129614    4477 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 22:06:56.145553 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129617    4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 22:06:56.145553 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129619    4477 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 22:06:56.145553 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129622    4477 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 22:06:56.145553 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129625    4477 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 22:06:56.145553 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129627    4477 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 22:06:56.145553 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129631    4477 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 22:06:56.145553 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129633    4477 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 22:06:56.145553 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.129657    4477 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 22:06:56.145553 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129688    4477 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 22:06:56.145553 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129693    4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 22:06:56.146418 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129696    4477 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 22:06:56.146418 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129699    4477 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 22:06:56.146418 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129702    4477 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 22:06:56.146418 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129706    4477 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 22:06:56.146418 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129709    4477 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 22:06:56.146418 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129711    4477 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 22:06:56.146418 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129714    4477 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 22:06:56.146418 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129717    4477 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 22:06:56.146418 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129719    4477 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 22:06:56.146418 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129722    4477 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 22:06:56.146418 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129725    4477 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 22:06:56.146418 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129727    4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 22:06:56.146418 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129730    4477 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 22:06:56.146418 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129733    4477 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 22:06:56.146418 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129735    4477 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 22:06:56.146418 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129738    4477 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 22:06:56.146418 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129740    4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 22:06:56.146418 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129743    4477 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 22:06:56.146418 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129746    4477 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 22:06:56.146418 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129748    4477 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 22:06:56.147290 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129751    4477 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 22:06:56.147290 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129753    4477 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 22:06:56.147290 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129756    4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 22:06:56.147290 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129758    4477 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 22:06:56.147290 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129776    4477 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 22:06:56.147290 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129779    4477 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 22:06:56.147290 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129782    4477 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 22:06:56.147290 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129785    4477 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 22:06:56.147290 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129788    4477 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 22:06:56.147290 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129791    4477 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 22:06:56.147290 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129793    4477 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 22:06:56.147290 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129796    4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 22:06:56.147290 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129799    4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 22:06:56.147290 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129801    4477 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 22:06:56.147290 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129804    4477 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 22:06:56.147290 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129808    4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 22:06:56.147290 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129811    4477 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 22:06:56.147290 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129814    4477 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 22:06:56.147290 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129816    4477 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 22:06:56.147290 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129819    4477 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 22:06:56.148155 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129821    4477 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 22:06:56.148155 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129824    4477 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 22:06:56.148155 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129827    4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 22:06:56.148155 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129829    4477 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 22:06:56.148155 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129832    4477 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 22:06:56.148155 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129835    4477 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 22:06:56.148155 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129837    4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 22:06:56.148155 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129840    4477 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 22:06:56.148155 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129842    4477 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 22:06:56.148155 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129845    4477 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 22:06:56.148155 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129848    4477 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 22:06:56.148155 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129851    4477 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 22:06:56.148155 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129853    4477 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 22:06:56.148155 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129856    4477 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 22:06:56.148155 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129859    4477 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 22:06:56.148155 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129861    4477 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 22:06:56.148155 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129864    4477 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 22:06:56.148155 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129866    4477 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 22:06:56.148155 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129869    4477 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 22:06:56.148155 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129872    4477 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 22:06:56.149033 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129874    4477 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 22:06:56.149033 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129877    4477 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 22:06:56.149033 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129880    4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 22:06:56.149033 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129883    4477 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 22:06:56.149033 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129886    4477 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 22:06:56.149033 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129889    4477 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 22:06:56.149033 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129891    4477 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 22:06:56.149033 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129895    4477 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 22:06:56.149033 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129897    4477 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 22:06:56.149033 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129900    4477 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 22:06:56.149033 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129903    4477 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 22:06:56.149033 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129905    4477 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 22:06:56.149033 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129908    4477 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 22:06:56.149033 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129910    4477 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 22:06:56.149033 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129913    4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 22:06:56.149033 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129915    4477 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 22:06:56.149033 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129918    4477 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 22:06:56.149033 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129921    4477 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 22:06:56.149033 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129923    4477 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 22:06:56.149872 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129926    4477 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 22:06:56.149872 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129928    4477 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 22:06:56.149872 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129931    4477 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 22:06:56.149872 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129933    4477 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 22:06:56.149872 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129937    4477 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 22:06:56.149872 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129941    4477 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 22:06:56.149872 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129944    4477 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 22:06:56.149872 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129948    4477 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 22:06:56.149872 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129951    4477 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 22:06:56.149872 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129953    4477 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 22:06:56.149872 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129956    4477 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 22:06:56.149872 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129958    4477 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 22:06:56.149872 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129961    4477 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 22:06:56.149872 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129964    4477 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 22:06:56.149872 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129966    4477 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 22:06:56.149872 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129969    4477 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 22:06:56.149872 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129972    4477 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 22:06:56.149872 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129974    4477 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 22:06:56.149872 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129977    4477 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 22:06:56.150422 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129979    4477 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 22:06:56.150422 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129984    4477 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 22:06:56.150422 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129988    4477 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 22:06:56.150422 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129992    4477 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 22:06:56.150422 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129995    4477 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 22:06:56.150422 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:56.129997    4477 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 22:06:56.150422 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.130068    4477 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 22:06:56.150422 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.130975    4477 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 30 22:06:56.150422 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.131023    4477 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 30 22:06:56.150422 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.131221    4477 server.go:1014] "Starting client certificate rotation"
Oct 30 22:06:56.150422 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.131361    4477 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 22:06:56.150422 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.131411    4477 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 22:01:16 +0000 UTC" deadline="2028-04-29 06:20:37.426652565 +0000 UTC"
Oct 30 22:06:56.150422 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.131437    4477 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13112h13m41.295217634s"
Oct 30 22:06:56.150422 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.131678    4477 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 22:06:56.150422 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.132702    4477 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 22:06:56.150422 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.134018    4477 log.go:25] "Validated CRI v1 runtime API"
Oct 30 22:06:56.150849 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.135526    4477 log.go:25] "Validated CRI v1 image API"
Oct 30 22:06:56.150849 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.136320    4477 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 22:06:56.150849 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.137626    4477 fs.go:135] Filesystem UUIDs: map[6f6aa058-cd52-470d-89c7-a1636208e0c0:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 a37948a0-7809-47bd-97e0-9bbfa84028b6:/dev/nvme0n1p4]
Oct 30 22:06:56.150937 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.137646    4477 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/26201244dde3aa78555ef013f144cecdb7b69913fb0d52e2e8674b4bbad190b5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/26201244dde3aa78555ef013f144cecdb7b69913fb0d52e2e8674b4bbad190b5/userdata/shm major:0 minor:220 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/285e3b8fbd1b072ec18318062c4f11065437f96d1131aaf0ec321e4aa459a467/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/285e3b8fbd1b072ec18318062c4f11065437f96d1131aaf0ec321e4aa459a467/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/328a6e8e630e43c5536d50c0c617888a400f575042f78833b718b6ab8188c58d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/328a6e8e630e43c5536d50c0c617888a400f575042f78833b718b6ab8188c58d/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/45efc55eedbfd6c64303396ae3d213a3b6cf4e9f53fc0a0bab6ab516227fe3c5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/45efc55eedbfd6c64303396ae3d213a3b6cf4e9f53fc0a0bab6ab516227fe3c5/userdata/shm major:0 minor:81 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6a85dcbaf30c440b28905b089dc95fc692b8a0ae5b68608d48f04db27b1ba940/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6a85dcbaf30c440b28905b089dc95fc692b8a0ae5b68608d48f04db27b1ba940/userdata/shm major:0 minor:218 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/755aed24fd1307c4e0126a7576b17171fbcc4ec7d80432a009149af17d0d3049/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/755aed24fd1307c4e0126a7576b17171fbcc4ec7d80432a009149af17d0d3049/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d71ca6f17b221e257084adf9799d36cca85cac57bf35ba31239f5bd9f67fff97/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d71ca6f17b221e257084adf9799d36cca85cac57bf35ba31239f5bd9f67fff97/userdata/shm major:0 minor:214 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/da9614088a069dcc12ff6b8f4b299e40bbcfa0bf887bc17decc6d7f541d3723b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/da9614088a069dcc12ff6b8f4b299e40bbcfa0bf887bc17decc6d7f541d3723b/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e0346fe028ed362ec3b464577562de2ae63214a8eaf5f062879fd52f0b3ca089/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e0346fe028ed362ec3b464577562de2ae63214a8eaf5f062879fd52f0b3ca089/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/eaf4f0878c59e7dec85bba1fc9a00bd5871a054e5b82711075cba359d8242920/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/eaf4f0878c59e7dec85bba1fc9a00bd5871a054e5b82711075cba359d8242920/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f36f29cca49b4d9575a65e7103b6b62995d203eaedddd57f3760c5ae248b025e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f36f29cca49b4d9575a65e7103b6b62995d203eaedddd57f3760c5ae248b025e/userdata/shm major:0 minor:202 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f47cd69ac5da515bb68e1c56a99c33e22b32498bf9de466ca92af4f71088dae2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f47cd69ac5da515bb68e1c56a99c33e22b32498bf9de466ca92af4f71088dae2/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f93b6fcd91ff7bdca9eb5ca4eb05183ad2ac1e9b0da47e40d0bc8be3748fd2ec/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f93b6fcd91ff7bdca9eb5ca4eb05183ad2ac1e9b0da47e40d0bc8be3748fd2ec/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/faaad9c80829f8bc4d3a4972a649291ae22f1ad8792b1393f1aef56194fb3f56/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/faaad9c80829f8bc4d3a4972a649291ae22f1ad8792b1393f1aef56194fb3f56/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/015f218d-bf1f-4e7e-bb5d-b805af003246/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/015f218d-bf1f-4e7e-bb5d-b805af003246/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/015f218d-bf1f-4e7e-bb5d-b805af003246/volumes/kubernetes.io~projected/kube-api-access-57q7f:{mountpoint:/var/lib/kubelet/pods/015f218d-bf1f-4e7e-bb5d-b805af003246/volumes/kubernetes.io~projected/kube-api-access-57q7f major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/015f218d-bf1f-4e7e-bb5d-b805af003246/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/015f218d-bf1f-4e7e-bb5d-b805af003246/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0e128ead-13e5-41f3-9ae7-6f944b5f18ee/volumes/kubernetes.io~projected/kube-api-access-mn4fl:{mountpoint:/var/lib/kubelet/pods/0e128ead-13e5-41f3-9ae7-6f944b5f18ee/volumes/kubernetes.io~projected/kube-api-access-mn4fl major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/15e3a459-855f-4181-825c-345e9b2e62e1/volumes/kubernetes.io~projected/kube-api-access-4ncxp:{mountpoint:/var/lib/kubelet/pods/15e3a459-855f-4181-825c-345e9b2e62e1/volumes/kubernetes.io~projected/kube-api-access-4ncxp major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2e4eaf5f-8a83-4681-b040-65676affdf7e/volumes/kubernetes.io~projected/kube-api-access-tfxls:{mountpoint:/var/lib/kubelet/pods/2e4eaf5f-8a83-4681-b040-65676affdf7e/volumes/kubernetes.io~projected/kube-api-access-tfxls major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7b50e755-6b0b-4d45-b7d3-c822127e6cb1/volumes/kubernetes.io~projected/kube-api-access-5d9bb:{mountpoint:/var/lib/kubelet/pods/7b50e755-6b0b-4d45-b7d3-c822127e6cb1/volumes/kubernetes.io~projected/kube-api-access-5d9bb major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/87f77489-c825-4f2c-a6ce-70e06f68f187/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/87f77489-c825-4f2c-a6ce-70e06f68f187/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/87f77489-c825-4f2c-a6ce-70e06f68f187/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/87f77489-c825-4f2c-a6ce-70e06f68f187/volumes/kubernetes.io~empty-dir/tmp major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/87f77489-c825-4f2c-a6ce-70e06f68f187/volumes/kubernetes.io~projected/kube-api-access-hc5rj:{mountpoint:/var/lib/kubelet/pods/87f77489-c825-4f2c-a6ce-70e06f68f187/volumes/kubernetes.io~projected/kube-api-access-hc5rj major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/98075172-cd56-41f2-8203-fc2639e95df9/volumes/kubernetes.io~projected/kube-api-access-8fngn:{mountpoint:/var/lib/kubelet/pods/98075172-cd56-41f2-8203-fc2639e95df9/volumes/kubernetes.io~projected/kube-api-access-8fngn major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/98075172-cd56-41f2-8203-fc2639e95df9/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/98075172-cd56-41f2-8203-fc2639e95df9/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9ea63ef4-6b84-46e6-a753-bca60280734a/volumes/kubernetes.io~projected/kube-api-access-ckg9z:{mountpoint:/var/lib/kubelet/pods/9ea63ef4-6b84-46e6-a753-bca60280734a/volumes/kubernetes.io~projected/kube-api-access-ckg9z major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a3d96078-3ec8-40aa-855a-d1ba4acf2e74/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/a3d96078-3ec8-40aa-855a-d1ba4acf2e74/volumes/kubernetes.io~secret/agent-certs major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b529dc48-9c57-4a0e-a4c0-49f53556ae34/volumes/kubernetes.io~projected/kube-api-access-2p2x4:{mountpoint:/var/lib/kubelet/pods/b529dc48-9c57-4a0e-a4c0-49f53556ae34/volumes/kubernetes.io~projected/kube-api-access-2p2x4 major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b8503b30-31d6-4454-a3fd-8bea4a527b04/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/b8503b30-31d6-4454-a3fd-8bea4a527b04/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d4e7cd84-8a3f-4ef7-970c-008a9c895a47/volumes/kubernetes.io~projected/kube-api-access-5d6l6:{mountpoint:/var/lib/kubelet/pods/d4e7cd84-8a3f-4ef7-970c-008a9c895a47/volumes/kubernetes.io~projected/kube-api-access-5d6l6 major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/da9b75bf-8af2-450f-9a98-c265eea48282/volumes/kubernetes.io~projected/kube-api-access-mkshh:{mountpoint:/var/lib/kubelet/pods/da9b75bf-8af2-450f-9a98-c265eea48282/volumes/kubernetes.io~projected/kube-api-access-mkshh major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dbf6b14c-5576-4ae3-bbb3-b3201aa7cb7c/volumes/kubernetes.io~projected/kube-api-access-8286n:{mountpoint:/var/lib/kubelet/pods/dbf6b14c-5576-4ae3-bbb3-b3201aa7cb7c/volumes/kubernetes.io~projected/kube-api-access-8286n major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e9c20fdf-bea5-4d6c-b252-b29b08bfdf63/volumes/kubernetes.io~projected/kube-api-access-5cdcp:{mountpoint:/var/lib/kubelet/pods/e9c20fdf-bea5-4d6c-b252-b29b08bfdf63/volumes/kubernetes.io~projected/kube-api-access-5cdcp major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fe4c45fd-690b-4111-9d15-93dc183b3e46/volumes/kubernetes.io~projected/kube-api-access-frxsh:{mountpoint:/var/lib/kubelet/pods/fe4c45fd-690b-4111-9d15-93dc183b3e46/volumes/kubernetes.io~projected/kube-api-access-frxsh major:0 minor:63 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/deec6a1a073295d3a693da356ce6381cae0eb6a2d12450446d8da668238a3468/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/9f79c50f2d95e48abbd5938c1beb976ebe53a141d55d6502bb037c10b8021878/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/e293b76cd3c7b09aa9f94f425b189da1073006c62fe8953ef75704a8dcf5e5c3/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/624193fa415267c74671ab7fcff129a7ff0c479fd70dbe73e0e568150565afc1/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/06bc5fdc58c362122b23a2e5907b864c891b5196237b3c70281c3a70a7458fd7/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/97312aa544dda410a05863e35e7412c6e581c429b22fa5eaa08df747660d59dc/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/eaa28678b19d9263ef91a994ff1adc4e99de02095102040feefb296995022aab/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/e20aaeb92fa3877c333c6df492680b2aa320e4e51f98254d25a508aeb59b4133/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/23c95602edc9cc03bf60fd342380d9e67f9978be4f70889ad70e267ed29458ee/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-136:{mountpoint:/var/lib/containers/storage/overlay/c561aec7a7d21ad7118481063d17ee31cc30332f07d032b2bb21a6b9368dcd3b/merged major:0 minor:136 fsType:overlay blockSize:0} overlay_0-138:{mountpoint:/var/lib/containers/storage/overlay/014a3bc8a89fca040cd3c768e4d761bfd314cc7599eee4c423ec810cc3b9f680/merged major:0 minor:138 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/7b7c20d4ecc31345e9125c39f0074180e1930d455c24e9d007457580136aff48/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-149:{mountpoint:/var/lib/containers/storage/overlay/7b3ff579835f36ded9c1de8e8ee469a0ad8f6a4a4b3935507c75417d0094105e/merged major:0 minor:149 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/8e754a9c2979af0181c12ab1c8b89ca2347c39001bcc677b37d3a1c135b6db73/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-159:{mountpoint:/var/lib/containers/storage/overlay/a1026a9499325d4b2ee325b2d46c51a222ca0ce24c6362c3c5ddf6108f114541/merged major:0 minor:159 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/430366b4cbe7ed968be1f287afb7ec0091bc52870a05af160ecbcaf5b41c78cc/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/2aee3377a4d8560bd883908210da4daef3b4dbb6baa8e954c162a8b9d035cd1d/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/5fd4bbd9180d1b42cd7754fdb89cf3607134b0dd489fa9906f11a0128fe524ea/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/094183a60a82798633c4f5c28664da529df6c04503536a9f2d5d964f5ada246e/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/d6578aec1eb7455d554419ab39fba446a96f9c39301bdc4cae2b2492eee59254/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/3ead02b103badb7743a38cc7e8f10b5cbc6ae19b1d06dcd224acb7b57e6175ea/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-204:{mountpoint:/var/lib/containers/storage/overlay/c31c12bef37060081c5a87883d83f69e74dd1ba55459f1e3a2f230cd6f71cbc4/merged major:0 minor:204 fsType:overlay blockSize:0} overlay_0-206:{mountpoint:/var/lib/containers/storage/overlay/c862cbd8796965bd62c55b0be8c1bcc694c3cbcd17ea630c12d4b6e18afde3b4/merged major:0 minor:206 fsType:overlay blockSize:0} overlay_0-222:{mountpoint:/var/lib/containers/storage/overlay/ab3002fa47447029ffc471accc07d20a961b6ec6da807fb6947e1f82ecc0c9ac/merged major:0 minor:222 fsType:overlay blockSize:0} overlay_0-224:{mountpoint:/var/lib/containers/storage/overlay/a0bc14000e811869c40ff6503e8aaced4c8da3f1ca3b6268a474b75691131bae/merged major:0 minor:224 fsType:overlay blockSize:0} overlay_0-226:{mountpoint:/var/lib/containers/storage/overlay/dbae7847b2d6c04a7fc38a3e0d221a196667699d4df5d7b90c068342cc11d828/merged major:0 minor:226 fsType:overlay blockSize:0} overlay_0-228:{mountpoint:/var/lib/containers/storage/overlay/723cf0a5289a4163f8cf7af3619683fe085f440340c0763de1e120dd0394f36f/merged major:0 minor:228 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/647a7196fbbda5b34efbf2fb1e8a9346381a37193f84cb4c893519a782d8245a/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-236:{mountpoint:/var/lib/containers/storage/overlay/41baa45df826e57e7c18c414ede042a552a2d744bae23f24c911beb851e76459/merged major:0 minor:236 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/e9cedc470f30c9ac1489c13d94f1f09a65f6df1bde768b204634f2f9bc3321b7/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/952c7e5de6246ae293fbade07132e39e35941c9daba070593cf71fa390ca1133/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/99626769650a207127cc4762b9639d321a759ac468f04ba8bdd4a6c6bf075695/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/11812463ab56e4937b11950d6a09a73174ef7fd420d84c7c0b280f92a836fc59/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/4f6ac7f2b5fc72f4dff7214426c109fc9e5b35dc4b3338ceec6480bc71a3f779/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/6d53912b85253048daa530608856a58e1dcf201744afd1f915ac7c58cfacd237/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/50c40b735eebe9612b2d5eb2fd418afdc05f71164346c4eeaff42d9229113269/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/4858986d902712ae6c56095b50fc2822f21cfeba00cd4716ea8bd14d44ff6ac6/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/84677d64cc449d462e4d7bf4bf2307fb59d8066402872f45d854e866ad7c5c80/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/94e6bb90a00a2ca03eb239f28b8e81987b0b82196fae6ce79fb55518b629e48c/merged major:0 minor:96 fsType:overlay blockSize:0}]
Oct 30 22:06:56.151095 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.143542    4477 manager.go:217] Machine: {Timestamp:2026-10-30 22:06:56.143083778 +0000 UTC m=+0.069009975 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099792 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2aaa8b88dcc89bc14401779433ff7e SystemUUID:ec2aaa8b-88dc-c89b-c144-01779433ff7e BootID:2d4fffbf-f706-4c4d-be0a-dc4764b67222 Filesystems:[{Device:/run/containers/storage/overlay-containers/faaad9c80829f8bc4d3a4972a649291ae22f1ad8792b1393f1aef56194fb3f56/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b8503b30-31d6-4454-a3fd-8bea4a527b04/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:208 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-236 DeviceMajor:0 DeviceMinor:236 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-226 DeviceMajor:0 DeviceMinor:226 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-228 DeviceMajor:0 DeviceMinor:228 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9ea63ef4-6b84-46e6-a753-bca60280734a/volumes/kubernetes.io~projected/kube-api-access-ckg9z DeviceMajor:0 DeviceMinor:201 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-204 DeviceMajor:0 DeviceMinor:204 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7b50e755-6b0b-4d45-b7d3-c822127e6cb1/volumes/kubernetes.io~projected/kube-api-access-5d9bb DeviceMajor:0 DeviceMinor:59 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/015f218d-bf1f-4e7e-bb5d-b805af003246/volumes/kubernetes.io~projected/kube-api-access-57q7f DeviceMajor:0 DeviceMinor:67 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/87f77489-c825-4f2c-a6ce-70e06f68f187/volumes/kubernetes.io~projected/kube-api-access-hc5rj DeviceMajor:0 DeviceMinor:61 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/dbf6b14c-5576-4ae3-bbb3-b3201aa7cb7c/volumes/kubernetes.io~projected/kube-api-access-8286n DeviceMajor:0 DeviceMinor:217 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/15e3a459-855f-4181-825c-345e9b2e62e1/volumes/kubernetes.io~projected/kube-api-access-4ncxp DeviceMajor:0 DeviceMinor:65 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0e128ead-13e5-41f3-9ae7-6f944b5f18ee/volumes/kubernetes.io~projected/kube-api-access-mn4fl DeviceMajor:0 DeviceMinor:66 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/328a6e8e630e43c5536d50c0c617888a400f575042f78833b718b6ab8188c58d/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/285e3b8fbd1b072ec18318062c4f11065437f96d1131aaf0ec321e4aa459a467/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/87f77489-c825-4f2c-a6ce-70e06f68f187/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:57 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/da9b75bf-8af2-450f-9a98-c265eea48282/volumes/kubernetes.io~projected/kube-api-access-mkshh DeviceMajor:0 DeviceMinor:64 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-138 DeviceMajor:0 DeviceMinor:138 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-206 DeviceMajor:0 DeviceMinor:206 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/755aed24fd1307c4e0126a7576b17171fbcc4ec7d80432a009149af17d0d3049/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-136 DeviceMajor:0 DeviceMinor:136 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/run/containers/storage/overlay-containers/da9614088a069dcc12ff6b8f4b299e40bbcfa0bf887bc17decc6d7f541d3723b/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-222 DeviceMajor:0 DeviceMinor:222 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a3d96078-3ec8-40aa-855a-d1ba4acf2e74/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:56 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f47cd69ac5da515bb68e1c56a99c33e22b32498bf9de466ca92af4f71088dae2/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d4e7cd84-8a3f-4ef7-970c-008a9c895a47/volumes/kubernetes.io~projected/kube-api-access-5d6l6 DeviceMajor:0 DeviceMinor:216 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/eaf4f0878c59e7dec85bba1fc9a00bd5871a054e5b82711075cba359d8242920/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f36f29cca49b4d9575a65e7103b6b62995d203eaedddd57f3760c5ae248b025e/userdata/shm DeviceMajor:0 DeviceMinor:202 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b529dc48-9c57-4a0e-a4c0-49f53556ae34/volumes/kubernetes.io~projected/kube-api-access-2p2x4 DeviceMajor:0 DeviceMinor:209 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fe4c45fd-690b-4111-9d15-93dc183b3e46/volumes/kubernetes.io~projected/kube-api-access-frxsh DeviceMajor:0 DeviceMinor:63 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-149 DeviceMajor:0 DeviceMinor:149 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d71ca6f17b221e257084adf9799d36cca85cac57bf35ba31239f5bd9f67fff97/userdata/shm DeviceMajor:0 DeviceMinor:214 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f93b6fcd91ff7bdca9eb5ca4eb05183ad2ac1e9b0da47e40d0bc8be3748fd2ec/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/26201244dde3aa78555ef013f144cecdb7b69913fb0d52e2e8674b4bbad190b5/userdata/shm DeviceMajor:0 DeviceMinor:220 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e0346fe028ed362ec3b464577562de2ae63214a8eaf5f062879fd52f0b3ca089/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/45efc55eedbfd6c64303396ae3d213a3b6cf4e9f53fc0a0bab6ab516227fe3c5/userdata/shm DeviceMajor:0 DeviceMinor:81 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-224 DeviceMajor:0 DeviceMinor:224 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-159 DeviceMajor:0 DeviceMinor:159 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/98075172-cd56-41f2-8203-fc2639e95df9/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6a85dcbaf30c440b28905b089dc95fc692b8a0ae5b68608d48f04db27b1ba940/userdata/shm DeviceMajor:0 DeviceMinor:218 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/015f218d-bf1f-4e7e-bb5d-b805af003246/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:55 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2e4eaf5f-8a83-4681-b040-65676affdf7e/volumes/kubernetes.io~projected/kube-api-access-tfxls DeviceMajor:0 DeviceMinor:60 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e9c20fdf-bea5-4d6c-b252-b29b08bfdf63/volumes/kubernetes.io~projected/kube-api-access-5cdcp DeviceMajor:0 DeviceMinor:62 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/015f218d-bf1f-4e7e-bb5d-b805af003246/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/98075172-cd56-41f2-8203-fc2639e95df9/volumes/kubernetes.io~projected/kube-api-access-8fngn DeviceMajor:0 DeviceMinor:197 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/87f77489-c825-4f2c-a6ce-70e06f68f187/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:58 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:26201244dde3aa7 MacAddress:22:64:5a:5e:28:d0 Speed:10000 Mtu:8901} {Name:6a85dcbaf30c440 MacAddress:8a:22:be:b9:13:f9 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:f6:63:53:65 Speed:0 Mtu:9001} {Name:br-int MacAddress:02:99:84:4f:50:5d Speed:0 Mtu:8901} {Name:d71ca6f17b221e2 MacAddress:7e:3e:76:6b:1c:41 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:f6:63:53:65 Speed:0 Mtu:9001} {Name:f36f29cca49b4d9 MacAddress:86:79:3d:4d:5c:4e Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:de:cb:84:a8:02:1c Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:72:25:0f:ca:09:9f Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 22:06:56.151095 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.143756    4477 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 22:06:56.151095 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.143793    4477 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 22:06:56.151095 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.144327    4477 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 22:06:56.151095 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.144352    4477 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-77.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 22:06:56.151095 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.144667    4477 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 22:06:56.151095 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.144680    4477 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 22:06:56.151095 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.144694    4477 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 22:06:56.151095 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.144721    4477 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 22:06:56.151095 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.144832    4477 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 22:06:56.151095 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.145110    4477 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 22:06:56.151095 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.145374    4477 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 22:06:56.151095 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.145421    4477 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 22:06:56.151095 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.145439    4477 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 22:06:56.151095 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.145453    4477 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 22:06:56.151095 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.145465    4477 kubelet.go:405] "Adding apiserver pod source"
Oct 30 22:06:56.151095 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.145476    4477 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 22:06:56.151095 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.145993    4477 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 22:06:56.151095 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.146006    4477 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 22:06:56.151095 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.146420    4477 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 22:06:56.151095 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.148172    4477 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 22:06:56.151095 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.148182    4477 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 22:06:56.151095 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.148275    4477 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 30 22:06:56.151095 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.148465    4477 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 22:06:56.151095 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.148493    4477 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 22:06:56.151095 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.148602    4477 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 22:06:56.151095 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.148634    4477 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 22:06:56.151095 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.148646    4477 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 22:06:56.151095 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.148657    4477 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 22:06:56.151095 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.148669    4477 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 22:06:56.151095 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.148680    4477 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 22:06:56.151095 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.148693    4477 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 22:06:56.151095 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.148702    4477 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 22:06:56.151095 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.148718    4477 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 22:06:56.151095 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.148724    4477 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 22:06:56.151095 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.148739    4477 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 22:06:56.151095 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.148748    4477 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 22:06:56.151095 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.148772    4477 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 22:06:56.151095 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.148782    4477 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 22:06:56.152261 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.151952    4477 server.go:1263] "Started kubelet"
Oct 30 22:06:56.152261 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.151993    4477 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 22:06:56.152588 ip-10-0-130-77 systemd[1]: Started Kubernetes Kubelet.
Oct 30 22:06:56.154724 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.152457    4477 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 22:06:56.154724 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.152763    4477 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 22:06:56.154724 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.153140    4477 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 22:06:56.154724 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.153891    4477 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 22:06:56.154724 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.154697    4477 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 22:06:56.154960 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.154740    4477 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 22:06:56.154960 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.154793    4477 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 22:01:18 +0000 UTC" deadline="2028-05-13 05:35:53.059150645 +0000 UTC"
Oct 30 22:06:56.154960 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.154822    4477 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13447h28m56.90433133s"
Oct 30 22:06:56.155865 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:06:56.154941    4477 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-77.ec2.internal\" not found"
Oct 30 22:06:56.155917 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.155903    4477 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 22:06:56.155966 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.155915    4477 factory.go:55] Registering systemd factory
Oct 30 22:06:56.155966 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.155939    4477 factory.go:223] Registration of the systemd container factory successfully
Oct 30 22:06:56.156306 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.156287    4477 factory.go:153] Registering CRI-O factory
Oct 30 22:06:56.156306 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.156308    4477 factory.go:223] Registration of the crio container factory successfully
Oct 30 22:06:56.156485 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.156334    4477 factory.go:103] Registering Raw factory
Oct 30 22:06:56.156485 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.156354    4477 manager.go:1199] Started watching for new ooms in manager
Oct 30 22:06:56.156662 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.156651    4477 manager.go:319] Starting recovery of all containers
Oct 30 22:06:56.157870 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.157586    4477 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 22:06:56.157870 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.157602    4477 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 22:06:56.157870 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.157708    4477 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 22:06:56.161494 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.161139    4477 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="015f218d-bf1f-4e7e-bb5d-b805af003246" volumeName="kubernetes.io/configmap/015f218d-bf1f-4e7e-bb5d-b805af003246-env-overrides" seLinuxMountContext=""
Oct 30 22:06:56.161494 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.161203    4477 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="87f77489-c825-4f2c-a6ce-70e06f68f187" volumeName="kubernetes.io/empty-dir/87f77489-c825-4f2c-a6ce-70e06f68f187-tmp" seLinuxMountContext=""
Oct 30 22:06:56.161494 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.161245    4477 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="98075172-cd56-41f2-8203-fc2639e95df9" volumeName="kubernetes.io/projected/98075172-cd56-41f2-8203-fc2639e95df9-kube-api-access-8fngn" seLinuxMountContext=""
Oct 30 22:06:56.161494 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.161260    4477 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="15e3a459-855f-4181-825c-345e9b2e62e1" volumeName="kubernetes.io/configmap/15e3a459-855f-4181-825c-345e9b2e62e1-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 30 22:06:56.161494 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.161294    4477 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a3d96078-3ec8-40aa-855a-d1ba4acf2e74" volumeName="kubernetes.io/secret/a3d96078-3ec8-40aa-855a-d1ba4acf2e74-agent-certs" seLinuxMountContext=""
Oct 30 22:06:56.161494 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.161314    4477 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="da9b75bf-8af2-450f-9a98-c265eea48282" volumeName="kubernetes.io/projected/da9b75bf-8af2-450f-9a98-c265eea48282-kube-api-access-mkshh" seLinuxMountContext=""
Oct 30 22:06:56.161494 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.161328    4477 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e9c20fdf-bea5-4d6c-b252-b29b08bfdf63" volumeName="kubernetes.io/projected/e9c20fdf-bea5-4d6c-b252-b29b08bfdf63-kube-api-access-5cdcp" seLinuxMountContext=""
Oct 30 22:06:56.161494 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.161458    4477 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="87f77489-c825-4f2c-a6ce-70e06f68f187" volumeName="kubernetes.io/empty-dir/87f77489-c825-4f2c-a6ce-70e06f68f187-etc-tuned" seLinuxMountContext=""
Oct 30 22:06:56.161903 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.161500    4477 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0e128ead-13e5-41f3-9ae7-6f944b5f18ee" volumeName="kubernetes.io/projected/0e128ead-13e5-41f3-9ae7-6f944b5f18ee-kube-api-access-mn4fl" seLinuxMountContext=""
Oct 30 22:06:56.161903 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.161520    4477 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2e4eaf5f-8a83-4681-b040-65676affdf7e" volumeName="kubernetes.io/configmap/2e4eaf5f-8a83-4681-b040-65676affdf7e-iptables-alerter-script" seLinuxMountContext=""
Oct 30 22:06:56.161903 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.161531    4477 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7b50e755-6b0b-4d45-b7d3-c822127e6cb1" volumeName="kubernetes.io/configmap/7b50e755-6b0b-4d45-b7d3-c822127e6cb1-cni-binary-copy" seLinuxMountContext=""
Oct 30 22:06:56.161903 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.161566    4477 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b529dc48-9c57-4a0e-a4c0-49f53556ae34" volumeName="kubernetes.io/projected/b529dc48-9c57-4a0e-a4c0-49f53556ae34-kube-api-access-2p2x4" seLinuxMountContext=""
Oct 30 22:06:56.161903 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.161583    4477 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fe4c45fd-690b-4111-9d15-93dc183b3e46" volumeName="kubernetes.io/projected/fe4c45fd-690b-4111-9d15-93dc183b3e46-kube-api-access-frxsh" seLinuxMountContext=""
Oct 30 22:06:56.161903 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.161597    4477 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="015f218d-bf1f-4e7e-bb5d-b805af003246" volumeName="kubernetes.io/configmap/015f218d-bf1f-4e7e-bb5d-b805af003246-ovnkube-config" seLinuxMountContext=""
Oct 30 22:06:56.161903 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.161632    4477 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="015f218d-bf1f-4e7e-bb5d-b805af003246" volumeName="kubernetes.io/configmap/015f218d-bf1f-4e7e-bb5d-b805af003246-ovnkube-script-lib" seLinuxMountContext=""
Oct 30 22:06:56.161903 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.161645    4477 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="15e3a459-855f-4181-825c-345e9b2e62e1" volumeName="kubernetes.io/configmap/15e3a459-855f-4181-825c-345e9b2e62e1-cni-binary-copy" seLinuxMountContext=""
Oct 30 22:06:56.161903 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.161710    4477 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7b50e755-6b0b-4d45-b7d3-c822127e6cb1" volumeName="kubernetes.io/configmap/7b50e755-6b0b-4d45-b7d3-c822127e6cb1-multus-daemon-config" seLinuxMountContext=""
Oct 30 22:06:56.161903 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.161726    4477 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9ea63ef4-6b84-46e6-a753-bca60280734a" volumeName="kubernetes.io/configmap/9ea63ef4-6b84-46e6-a753-bca60280734a-config-volume" seLinuxMountContext=""
Oct 30 22:06:56.161903 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.161745    4477 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9ea63ef4-6b84-46e6-a753-bca60280734a" volumeName="kubernetes.io/projected/9ea63ef4-6b84-46e6-a753-bca60280734a-kube-api-access-ckg9z" seLinuxMountContext=""
Oct 30 22:06:56.161903 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.161759    4477 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b8503b30-31d6-4454-a3fd-8bea4a527b04" volumeName="kubernetes.io/secret/b8503b30-31d6-4454-a3fd-8bea4a527b04-original-pull-secret" seLinuxMountContext=""
Oct 30 22:06:56.161903 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.161775    4477 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="015f218d-bf1f-4e7e-bb5d-b805af003246" volumeName="kubernetes.io/projected/015f218d-bf1f-4e7e-bb5d-b805af003246-kube-api-access-57q7f" seLinuxMountContext=""
Oct 30 22:06:56.161903 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.161794    4477 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9ea63ef4-6b84-46e6-a753-bca60280734a" volumeName="kubernetes.io/empty-dir/9ea63ef4-6b84-46e6-a753-bca60280734a-tmp-dir" seLinuxMountContext=""
Oct 30 22:06:56.161903 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.161865    4477 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b529dc48-9c57-4a0e-a4c0-49f53556ae34" volumeName="kubernetes.io/empty-dir/b529dc48-9c57-4a0e-a4c0-49f53556ae34-data-volume" seLinuxMountContext=""
Oct 30 22:06:56.161903 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.161884    4477 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="015f218d-bf1f-4e7e-bb5d-b805af003246" volumeName="kubernetes.io/secret/015f218d-bf1f-4e7e-bb5d-b805af003246-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 30 22:06:56.161903 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.161898    4477 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="15e3a459-855f-4181-825c-345e9b2e62e1" volumeName="kubernetes.io/configmap/15e3a459-855f-4181-825c-345e9b2e62e1-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 30 22:06:56.162755 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.161971    4477 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="87f77489-c825-4f2c-a6ce-70e06f68f187" volumeName="kubernetes.io/projected/87f77489-c825-4f2c-a6ce-70e06f68f187-kube-api-access-hc5rj" seLinuxMountContext=""
Oct 30 22:06:56.162755 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.161990    4477 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a3d96078-3ec8-40aa-855a-d1ba4acf2e74" volumeName="kubernetes.io/configmap/a3d96078-3ec8-40aa-855a-d1ba4acf2e74-konnectivity-ca" seLinuxMountContext=""
Oct 30 22:06:56.162755 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.162078    4477 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d4e7cd84-8a3f-4ef7-970c-008a9c895a47" volumeName="kubernetes.io/projected/d4e7cd84-8a3f-4ef7-970c-008a9c895a47-kube-api-access-5d6l6" seLinuxMountContext=""
Oct 30 22:06:56.162755 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.162105    4477 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="da9b75bf-8af2-450f-9a98-c265eea48282" volumeName="kubernetes.io/empty-dir/da9b75bf-8af2-450f-9a98-c265eea48282-tmp-dir" seLinuxMountContext=""
Oct 30 22:06:56.162755 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.162191    4477 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="15e3a459-855f-4181-825c-345e9b2e62e1" volumeName="kubernetes.io/projected/15e3a459-855f-4181-825c-345e9b2e62e1-kube-api-access-4ncxp" seLinuxMountContext=""
Oct 30 22:06:56.162755 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.162210    4477 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2e4eaf5f-8a83-4681-b040-65676affdf7e" volumeName="kubernetes.io/projected/2e4eaf5f-8a83-4681-b040-65676affdf7e-kube-api-access-tfxls" seLinuxMountContext=""
Oct 30 22:06:56.162755 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.162222    4477 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7b50e755-6b0b-4d45-b7d3-c822127e6cb1" volumeName="kubernetes.io/projected/7b50e755-6b0b-4d45-b7d3-c822127e6cb1-kube-api-access-5d9bb" seLinuxMountContext=""
Oct 30 22:06:56.162755 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.162346    4477 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="98075172-cd56-41f2-8203-fc2639e95df9" volumeName="kubernetes.io/secret/98075172-cd56-41f2-8203-fc2639e95df9-cert" seLinuxMountContext=""
Oct 30 22:06:56.162755 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.162360    4477 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dbf6b14c-5576-4ae3-bbb3-b3201aa7cb7c" volumeName="kubernetes.io/projected/dbf6b14c-5576-4ae3-bbb3-b3201aa7cb7c-kube-api-access-8286n" seLinuxMountContext=""
Oct 30 22:06:56.162755 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.162377    4477 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e9c20fdf-bea5-4d6c-b252-b29b08bfdf63" volumeName="kubernetes.io/configmap/e9c20fdf-bea5-4d6c-b252-b29b08bfdf63-serviceca" seLinuxMountContext=""
Oct 30 22:06:56.162755 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.162393    4477 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 22:06:56.162755 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.162402    4477 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 22:06:56.171658 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.171632    4477 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jcf8l_015f218d-bf1f-4e7e-bb5d-b805af003246/ovn-acl-logging/0.log"
Oct 30 22:06:56.171988 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.171966    4477 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jcf8l_015f218d-bf1f-4e7e-bb5d-b805af003246/ovn-acl-logging/0.log"
Oct 30 22:06:56.180858 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.180818    4477 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 22:06:56.185584 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.185540    4477 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 22:06:56.185584 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.185575    4477 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 22:06:56.185775 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.185594    4477 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 22:06:56.185775 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:06:56.185632    4477 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 30 22:06:56.193865 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.193803    4477 generic.go:356] "Generic (PLEG): container finished" podID="19ea3e9d10869a3d260d4dc45edca0a4" containerID="f042eb44796cb6b63e036a0c4f37226bee45acf9ec48c3dc9086ed7eaa81e479" exitCode=0
Oct 30 22:06:56.199615 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.199582    4477 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 22:06:56.199713 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.199654    4477 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 30 22:06:56.200459 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.200442    4477 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 22:06:56.201280 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.201260    4477 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jcf8l_015f218d-bf1f-4e7e-bb5d-b805af003246/ovn-acl-logging/0.log"
Oct 30 22:06:56.201611 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.201593    4477 generic.go:356] "Generic (PLEG): container finished" podID="015f218d-bf1f-4e7e-bb5d-b805af003246" containerID="4d528146f7ae00a132df470b558b19deb09cbf4fb7b12143466a4aba4409814f" exitCode=1
Oct 30 22:06:56.211338 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.211317    4477 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 22:06:56.211616 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.211597    4477 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 22:06:56.219106 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.219081    4477 generic.go:356] "Generic (PLEG): container finished" podID="15e3a459-855f-4181-825c-345e9b2e62e1" containerID="7d09f456e8c9e6451509bbc44aacca9e936b21bd4a2b31aa7b67acf2e1ba6cb5" exitCode=0
Oct 30 22:06:56.219106 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.219106    4477 generic.go:356] "Generic (PLEG): container finished" podID="15e3a459-855f-4181-825c-345e9b2e62e1" containerID="5063ef5a1b132a05514d139322ab1915957775308c5b0e42b612db3072b4e958" exitCode=0
Oct 30 22:06:56.219214 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.219115    4477 generic.go:356] "Generic (PLEG): container finished" podID="15e3a459-855f-4181-825c-345e9b2e62e1" containerID="126d7083d31d188d3e448a6fd2bfc2eb9c8e63e854d43dcc9fb52cbb0d431239" exitCode=0
Oct 30 22:06:56.219214 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.219121    4477 generic.go:356] "Generic (PLEG): container finished" podID="15e3a459-855f-4181-825c-345e9b2e62e1" containerID="451659c940d92507abf61f860658709d783592769d5f73ee9f4633e9a81dbed4" exitCode=0
Oct 30 22:06:56.219214 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.219126    4477 generic.go:356] "Generic (PLEG): container finished" podID="15e3a459-855f-4181-825c-345e9b2e62e1" containerID="da8e5996fea0205110b62b279a7a2b5674fb4bedfd8d9c2a0ded60ba1e3b1446" exitCode=0
Oct 30 22:06:56.219214 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.219131    4477 generic.go:356] "Generic (PLEG): container finished" podID="15e3a459-855f-4181-825c-345e9b2e62e1" containerID="1405d0b5534998c510b75ae3dce5a799859b349d946be3c39d991cc54fb814e9" exitCode=0
Oct 30 22:06:56.225448 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.225434    4477 manager.go:324] Recovery completed
Oct 30 22:06:56.239890 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.239869    4477 cpu_manager.go:226] "Starting" policy="none"
Oct 30 22:06:56.239890 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.239886    4477 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 22:06:56.240019 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.239906    4477 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 22:06:56.240085 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.240024    4477 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 30 22:06:56.240085 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.240035    4477 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 30 22:06:56.240085 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.240062    4477 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 30 22:06:56.240085 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.240068    4477 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 30 22:06:56.240085 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.240073    4477 policy_none.go:50] "Start"
Oct 30 22:06:56.240085 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.240081    4477 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 22:06:56.240085 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.240089    4477 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 22:06:56.240294 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.240200    4477 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 30 22:06:56.240294 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.240207    4477 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 30 22:06:56.240294 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.240212    4477 policy_none.go:44] "Start"
Oct 30 22:06:56.244115 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.244091    4477 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 22:06:56.246638 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.246618    4477 manager.go:341] "Starting Device Plugin manager"
Oct 30 22:06:56.247080 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:06:56.246647    4477 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 22:06:56.247153 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.247082    4477 server.go:87] "Starting device plugin registration server"
Oct 30 22:06:56.247361 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.247347    4477 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 22:06:56.247424 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.247362    4477 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 22:06:56.247472 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.247452    4477 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 22:06:56.247566 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.247532    4477 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 22:06:56.247633 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.247574    4477 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 22:06:56.247633 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.247583    4477 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 22:06:56.247729 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.247614    4477 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T22:06:56.247563817Z","UUID":"5b803924-2f41-4f92-8f8d-5d305d118179","Handler":null,"Name":"","Endpoint":""}
Oct 30 22:06:56.248826 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:06:56.248805    4477 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 22:06:56.249000 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.248982    4477 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 22:06:56.249102 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.249060    4477 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 22:06:56.286146 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.286079    4477 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-77.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-77.ec2.internal"]
Oct 30 22:06:56.286673 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.286649    4477 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-77.ec2.internal"
Oct 30 22:06:56.286749 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.286681    4477 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-77.ec2.internal"
Oct 30 22:06:56.286749 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.286685    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-77.ec2.internal" event={"ID":"19ea3e9d10869a3d260d4dc45edca0a4","Type":"ContainerStarted","Data":"f1296a641251d40db80105088bb63befd770fa1cfd7d6d830455fb5566909dd3"}
Oct 30 22:06:56.286749 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.286733    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-77.ec2.internal" event={"ID":"19ea3e9d10869a3d260d4dc45edca0a4","Type":"ContainerDied","Data":"f042eb44796cb6b63e036a0c4f37226bee45acf9ec48c3dc9086ed7eaa81e479"}
Oct 30 22:06:56.286749 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.286749    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-77.ec2.internal" event={"ID":"19ea3e9d10869a3d260d4dc45edca0a4","Type":"ContainerStarted","Data":"755aed24fd1307c4e0126a7576b17171fbcc4ec7d80432a009149af17d0d3049"}
Oct 30 22:06:56.286865 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.286786    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-77.ec2.internal" event={"ID":"1929b84fce10dee760acfa96faf62b49","Type":"ContainerStarted","Data":"3746a9cd324d6d159937d32c5b63250f8da3be8ce7f3fedc055fab27b936a528"}
Oct 30 22:06:56.286865 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.286797    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-77.ec2.internal" event={"ID":"1929b84fce10dee760acfa96faf62b49","Type":"ContainerStarted","Data":"da9614088a069dcc12ff6b8f4b299e40bbcfa0bf887bc17decc6d7f541d3723b"}
Oct 30 22:06:56.309018 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.308996    4477 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 22:06:56.309113 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.309053    4477 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 22:06:56.309113 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:06:56.309089    4477 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-130-77.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-77.ec2.internal"
Oct 30 22:06:56.309213 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:06:56.309056    4477 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-130-77.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-77.ec2.internal"
Oct 30 22:06:56.347669 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.347651    4477 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 22:06:56.348675 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.348661    4477 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-77.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 22:06:56.348747 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.348686    4477 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-77.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 22:06:56.348747 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.348695    4477 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-77.ec2.internal" event="NodeHasSufficientPID"
Oct 30 22:06:56.348747 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.348719    4477 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-130-77.ec2.internal"
Oct 30 22:06:56.378464 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.378446    4477 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-130-77.ec2.internal"
Oct 30 22:06:56.378542 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.378488    4477 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-130-77.ec2.internal"
Oct 30 22:06:56.400753 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.400733    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/19ea3e9d10869a3d260d4dc45edca0a4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-77.ec2.internal\" (UID: \"19ea3e9d10869a3d260d4dc45edca0a4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-77.ec2.internal"
Oct 30 22:06:56.400825 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.400756    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/19ea3e9d10869a3d260d4dc45edca0a4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-77.ec2.internal\" (UID: \"19ea3e9d10869a3d260d4dc45edca0a4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-77.ec2.internal"
Oct 30 22:06:56.400825 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.400772    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1929b84fce10dee760acfa96faf62b49-config\") pod \"kube-apiserver-proxy-ip-10-0-130-77.ec2.internal\" (UID: \"1929b84fce10dee760acfa96faf62b49\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-77.ec2.internal"
Oct 30 22:06:56.501527 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.501504    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/19ea3e9d10869a3d260d4dc45edca0a4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-77.ec2.internal\" (UID: \"19ea3e9d10869a3d260d4dc45edca0a4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-77.ec2.internal"
Oct 30 22:06:56.501598 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.501529    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/19ea3e9d10869a3d260d4dc45edca0a4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-77.ec2.internal\" (UID: \"19ea3e9d10869a3d260d4dc45edca0a4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-77.ec2.internal"
Oct 30 22:06:56.501598 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.501546    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1929b84fce10dee760acfa96faf62b49-config\") pod \"kube-apiserver-proxy-ip-10-0-130-77.ec2.internal\" (UID: \"1929b84fce10dee760acfa96faf62b49\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-77.ec2.internal"
Oct 30 22:06:56.501598 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.501584    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1929b84fce10dee760acfa96faf62b49-config\") pod \"kube-apiserver-proxy-ip-10-0-130-77.ec2.internal\" (UID: \"1929b84fce10dee760acfa96faf62b49\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-77.ec2.internal"
Oct 30 22:06:56.501695 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.501615    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/19ea3e9d10869a3d260d4dc45edca0a4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-77.ec2.internal\" (UID: \"19ea3e9d10869a3d260d4dc45edca0a4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-77.ec2.internal"
Oct 30 22:06:56.501695 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:56.501615    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/19ea3e9d10869a3d260d4dc45edca0a4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-77.ec2.internal\" (UID: \"19ea3e9d10869a3d260d4dc45edca0a4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-77.ec2.internal"
Oct 30 22:06:57.146275 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.146240    4477 apiserver.go:52] "Watching apiserver"
Oct 30 22:06:57.181862 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.181827    4477 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 22:06:57.182274 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.182251    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-69blh","openshift-network-operator/iptables-alerter-vw5fg","kube-system/kube-apiserver-proxy-ip-10-0-130-77.ec2.internal","openshift-kube-storage-version-migrator/migrator-66df5d496b-nzx7v","kube-system/global-pull-secret-syncer-rjtc8","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bbkdz","openshift-network-diagnostics/network-check-target-vs5pt","openshift-ingress-canary/ingress-canary-wp8vp","openshift-dns/dns-default-c749j","openshift-image-registry/node-ca-b74lk","openshift-multus/multus-4s4vc","openshift-multus/multus-additional-cni-plugins-22c27","kube-system/konnectivity-agent-mm655","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-77.ec2.internal","openshift-cluster-node-tuning-operator/tuned-dr9c2","openshift-multus/network-metrics-daemon-6hhw9","openshift-dns/node-resolver-4knxq","openshift-ovn-kubernetes/ovnkube-node-jcf8l"]
Oct 30 22:06:57.185317 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.185296    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6hhw9"
Oct 30 22:06:57.188378 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.188359    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-c749j"
Oct 30 22:06:57.192019 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.191989    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-69blh"
Oct 30 22:06:57.200077 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.200057    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-pwrc4\""
Oct 30 22:06:57.200158 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.200075    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 22:06:57.200158 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.200151    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-7kw8g\""
Oct 30 22:06:57.202478 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.202461    4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 22:06:57.202525 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.202480    4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 22:06:57.202525 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.202469    4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 22:06:57.202588 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.202565    4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 22:06:57.202624 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.202595    4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 22:06:57.202658 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.202645    4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 22:06:57.207608 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.207590    4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 22:06:57.208330 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.208308    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 22:06:57.208422 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.208334    4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 22:06:57.208422 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.208346    4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 22:06:57.208422 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.208368    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-bf7kf\""
Oct 30 22:06:57.208422 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.208373    4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 22:06:57.208422 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.208403    4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 22:06:57.208599 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.208459    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-nn9g8\""
Oct 30 22:06:57.208599 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.208596    4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 22:06:57.208688 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.208671    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-k6z4j\""
Oct 30 22:06:57.208768 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.208693    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 22:06:57.208819 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.208804    4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 22:06:57.208819 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.208814    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 22:06:57.208916 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.208830    4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 22:06:57.208916 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.208898    4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 22:06:57.210683 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.210659    4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 22:06:57.210683 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.210666    4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 22:06:57.210802 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.210707    4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 22:06:57.210802 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.210715    4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 22:06:57.210802 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.210729    4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 22:06:57.210802 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.210729    4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 22:06:57.210802 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.210728    4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 22:06:57.210930 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.210833    4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 22:06:57.210996 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.210981    4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 22:06:57.211065 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.211035    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 22:06:57.211065 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.211060    4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 22:06:57.211143 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.211066    4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 22:06:57.211143 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.211059    4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 22:06:57.211203 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.211150    4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 22:06:57.211234 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.211217    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 22:06:57.213316 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.213303    4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 30 22:06:57.213508 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.213491    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 22:06:57.213571 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.213511    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 22:06:57.213571 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.213492    4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 30 22:06:57.213571 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.213545    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-c5qz2\""
Oct 30 22:06:57.213571 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.213555    4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 22:06:57.213571 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.213562    4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 22:06:57.213742 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.213516    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-ggpvh\""
Oct 30 22:06:57.213742 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.213592    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-44l7l\""
Oct 30 22:06:57.213742 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.213596    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-8wxtm\""
Oct 30 22:06:57.213742 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.213608    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-z6gtt\""
Oct 30 22:06:57.213742 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.213655    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-g5ww4\""
Oct 30 22:06:57.213742 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.213708    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-tjmjw\""
Oct 30 22:06:57.213891 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.213808    4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 22:06:57.214830 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.214817    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-26rrv\""
Oct 30 22:06:57.214917 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.214905    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-xwfpf\""
Oct 30 22:06:57.214955 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.214913    4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 22:06:57.214991 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.214975    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-rrq2z\""
Oct 30 22:06:57.215020 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.214916    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-htsvp\""
Oct 30 22:06:57.221853 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.221839    4477 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-77.ec2.internal"
Oct 30 22:06:57.221946 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.221935    4477 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-77.ec2.internal"
Oct 30 22:06:57.242825 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.242810    4477 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 22:06:57.242899 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:06:57.242850    4477 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-130-77.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-77.ec2.internal"
Oct 30 22:06:57.243594 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.243582    4477 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 22:06:57.243634 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:06:57.243610    4477 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-130-77.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-77.ec2.internal"
Oct 30 22:06:57.259191 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.259178    4477 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 22:06:57.305274 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.305256    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7b50e755-6b0b-4d45-b7d3-c822127e6cb1-multus-socket-dir-parent\") pod \"multus-4s4vc\" (UID: \"7b50e755-6b0b-4d45-b7d3-c822127e6cb1\") " pod="openshift-multus/multus-4s4vc"
Oct 30 22:06:57.305332 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.305279    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7b50e755-6b0b-4d45-b7d3-c822127e6cb1-host-run-netns\") pod \"multus-4s4vc\" (UID: \"7b50e755-6b0b-4d45-b7d3-c822127e6cb1\") " pod="openshift-multus/multus-4s4vc"
Oct 30 22:06:57.305332 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.305296    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/15e3a459-855f-4181-825c-345e9b2e62e1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-22c27\" (UID: \"15e3a459-855f-4181-825c-345e9b2e62e1\") " pod="openshift-multus/multus-additional-cni-plugins-22c27"
Oct 30 22:06:57.305332 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.305319    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b8503b30-31d6-4454-a3fd-8bea4a527b04-kubelet-config\") pod \"global-pull-secret-syncer-rjtc8\" (UID: \"b8503b30-31d6-4454-a3fd-8bea4a527b04\") " pod="kube-system/global-pull-secret-syncer-rjtc8"
Oct 30 22:06:57.305435 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.305393    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/015f218d-bf1f-4e7e-bb5d-b805af003246-node-log\") pod \"ovnkube-node-jcf8l\" (UID: \"015f218d-bf1f-4e7e-bb5d-b805af003246\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l"
Oct 30 22:06:57.305435 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.305419    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/015f218d-bf1f-4e7e-bb5d-b805af003246-ovnkube-script-lib\") pod \"ovnkube-node-jcf8l\" (UID: \"015f218d-bf1f-4e7e-bb5d-b805af003246\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l"
Oct 30 22:06:57.305510 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.305437    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/87f77489-c825-4f2c-a6ce-70e06f68f187-etc-modprobe-d\") pod \"tuned-dr9c2\" (UID: \"87f77489-c825-4f2c-a6ce-70e06f68f187\") " pod="openshift-cluster-node-tuning-operator/tuned-dr9c2"
Oct 30 22:06:57.305510 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.305451    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/87f77489-c825-4f2c-a6ce-70e06f68f187-etc-sysconfig\") pod \"tuned-dr9c2\" (UID: \"87f77489-c825-4f2c-a6ce-70e06f68f187\") " pod="openshift-cluster-node-tuning-operator/tuned-dr9c2"
Oct 30 22:06:57.305510 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.305466    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7b50e755-6b0b-4d45-b7d3-c822127e6cb1-etc-kubernetes\") pod \"multus-4s4vc\" (UID: \"7b50e755-6b0b-4d45-b7d3-c822127e6cb1\") " pod="openshift-multus/multus-4s4vc"
Oct 30 22:06:57.305510 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.305481    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/015f218d-bf1f-4e7e-bb5d-b805af003246-etc-openvswitch\") pod \"ovnkube-node-jcf8l\" (UID: \"015f218d-bf1f-4e7e-bb5d-b805af003246\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l"
Oct 30 22:06:57.305673 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.305540    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9ea63ef4-6b84-46e6-a753-bca60280734a-config-volume\") pod \"dns-default-c749j\" (UID: \"9ea63ef4-6b84-46e6-a753-bca60280734a\") " pod="openshift-dns/dns-default-c749j"
Oct 30 22:06:57.305673 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.305604    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5d6l6\" (UniqueName: \"kubernetes.io/projected/d4e7cd84-8a3f-4ef7-970c-008a9c895a47-kube-api-access-5d6l6\") pod \"migrator-66df5d496b-nzx7v\" (UID: \"d4e7cd84-8a3f-4ef7-970c-008a9c895a47\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-nzx7v"
Oct 30 22:06:57.305770 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.305674    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0e128ead-13e5-41f3-9ae7-6f944b5f18ee-etc-selinux\") pod \"aws-ebs-csi-driver-node-bbkdz\" (UID: \"0e128ead-13e5-41f3-9ae7-6f944b5f18ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bbkdz"
Oct 30 22:06:57.305770 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.305682    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/015f218d-bf1f-4e7e-bb5d-b805af003246-ovnkube-script-lib\") pod \"ovnkube-node-jcf8l\" (UID: \"015f218d-bf1f-4e7e-bb5d-b805af003246\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l"
Oct 30 22:06:57.305770 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.305694    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/87f77489-c825-4f2c-a6ce-70e06f68f187-run\") pod \"tuned-dr9c2\" (UID: \"87f77489-c825-4f2c-a6ce-70e06f68f187\") " pod="openshift-cluster-node-tuning-operator/tuned-dr9c2"
Oct 30 22:06:57.305770 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.305738    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/87f77489-c825-4f2c-a6ce-70e06f68f187-etc-tuned\") pod \"tuned-dr9c2\" (UID: \"87f77489-c825-4f2c-a6ce-70e06f68f187\") " pod="openshift-cluster-node-tuning-operator/tuned-dr9c2"
Oct 30 22:06:57.305770 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.305761    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hc5rj\" (UniqueName: \"kubernetes.io/projected/87f77489-c825-4f2c-a6ce-70e06f68f187-kube-api-access-hc5rj\") pod \"tuned-dr9c2\" (UID: \"87f77489-c825-4f2c-a6ce-70e06f68f187\") " pod="openshift-cluster-node-tuning-operator/tuned-dr9c2"
Oct 30 22:06:57.305998 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.305799    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7b50e755-6b0b-4d45-b7d3-c822127e6cb1-host-var-lib-cni-multus\") pod \"multus-4s4vc\" (UID: \"7b50e755-6b0b-4d45-b7d3-c822127e6cb1\") " pod="openshift-multus/multus-4s4vc"
Oct 30 22:06:57.305998 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.305817    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/87f77489-c825-4f2c-a6ce-70e06f68f187-etc-tuned\") pod \"tuned-dr9c2\" (UID: \"87f77489-c825-4f2c-a6ce-70e06f68f187\") " pod="openshift-cluster-node-tuning-operator/tuned-dr9c2"
Oct 30 22:06:57.305998 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.305825    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a3d96078-3ec8-40aa-855a-d1ba4acf2e74-konnectivity-ca\") pod \"konnectivity-agent-mm655\" (UID: \"a3d96078-3ec8-40aa-855a-d1ba4acf2e74\") " pod="kube-system/konnectivity-agent-mm655"
Oct 30 22:06:57.305998 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.305851    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7b50e755-6b0b-4d45-b7d3-c822127e6cb1-host-var-lib-cni-bin\") pod \"multus-4s4vc\" (UID: \"7b50e755-6b0b-4d45-b7d3-c822127e6cb1\") " pod="openshift-multus/multus-4s4vc"
Oct 30 22:06:57.305998 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.305873    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/98075172-cd56-41f2-8203-fc2639e95df9-cert\") pod \"ingress-canary-wp8vp\" (UID: \"98075172-cd56-41f2-8203-fc2639e95df9\") " pod="openshift-ingress-canary/ingress-canary-wp8vp"
Oct 30 22:06:57.305998 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.305879    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9ea63ef4-6b84-46e6-a753-bca60280734a-config-volume\") pod \"dns-default-c749j\" (UID: \"9ea63ef4-6b84-46e6-a753-bca60280734a\") " pod="openshift-dns/dns-default-c749j"
Oct 30 22:06:57.305998 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.305929    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0e128ead-13e5-41f3-9ae7-6f944b5f18ee-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bbkdz\" (UID: \"0e128ead-13e5-41f3-9ae7-6f944b5f18ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bbkdz"
Oct 30 22:06:57.305998 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.305967    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/87f77489-c825-4f2c-a6ce-70e06f68f187-etc-sysctl-d\") pod \"tuned-dr9c2\" (UID: \"87f77489-c825-4f2c-a6ce-70e06f68f187\") " pod="openshift-cluster-node-tuning-operator/tuned-dr9c2"
Oct 30 22:06:57.306303 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.306019    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/da9b75bf-8af2-450f-9a98-c265eea48282-hosts-file\") pod \"node-resolver-4knxq\" (UID: \"da9b75bf-8af2-450f-9a98-c265eea48282\") " pod="openshift-dns/node-resolver-4knxq"
Oct 30 22:06:57.306303 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.306090    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe4c45fd-690b-4111-9d15-93dc183b3e46-metrics-certs\") pod \"network-metrics-daemon-6hhw9\" (UID: \"fe4c45fd-690b-4111-9d15-93dc183b3e46\") " pod="openshift-multus/network-metrics-daemon-6hhw9"
Oct 30 22:06:57.306303 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.306118    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frxsh\" (UniqueName: \"kubernetes.io/projected/fe4c45fd-690b-4111-9d15-93dc183b3e46-kube-api-access-frxsh\") pod \"network-metrics-daemon-6hhw9\" (UID: \"fe4c45fd-690b-4111-9d15-93dc183b3e46\") " pod="openshift-multus/network-metrics-daemon-6hhw9"
Oct 30 22:06:57.306303 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.306182    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2e4eaf5f-8a83-4681-b040-65676affdf7e-iptables-alerter-script\") pod \"iptables-alerter-vw5fg\" (UID: \"2e4eaf5f-8a83-4681-b040-65676affdf7e\") " pod="openshift-network-operator/iptables-alerter-vw5fg"
Oct 30 22:06:57.306303 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.306219    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0e128ead-13e5-41f3-9ae7-6f944b5f18ee-sys-fs\") pod \"aws-ebs-csi-driver-node-bbkdz\" (UID: \"0e128ead-13e5-41f3-9ae7-6f944b5f18ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bbkdz"
Oct 30 22:06:57.306303 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.306247    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/87f77489-c825-4f2c-a6ce-70e06f68f187-sys\") pod \"tuned-dr9c2\" (UID: \"87f77489-c825-4f2c-a6ce-70e06f68f187\") " pod="openshift-cluster-node-tuning-operator/tuned-dr9c2"
Oct 30 22:06:57.306303 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.306253    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a3d96078-3ec8-40aa-855a-d1ba4acf2e74-konnectivity-ca\") pod \"konnectivity-agent-mm655\" (UID: \"a3d96078-3ec8-40aa-855a-d1ba4acf2e74\") " pod="kube-system/konnectivity-agent-mm655"
Oct 30 22:06:57.306303 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.306187    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/98075172-cd56-41f2-8203-fc2639e95df9-cert\") pod \"ingress-canary-wp8vp\" (UID: \"98075172-cd56-41f2-8203-fc2639e95df9\") " pod="openshift-ingress-canary/ingress-canary-wp8vp"
Oct 30 22:06:57.306303 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.306271    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2e4eaf5f-8a83-4681-b040-65676affdf7e-host-slash\") pod \"iptables-alerter-vw5fg\" (UID: \"2e4eaf5f-8a83-4681-b040-65676affdf7e\") " pod="openshift-network-operator/iptables-alerter-vw5fg"
Oct 30 22:06:57.306671 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.306321    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/015f218d-bf1f-4e7e-bb5d-b805af003246-host-cni-bin\") pod \"ovnkube-node-jcf8l\" (UID: \"015f218d-bf1f-4e7e-bb5d-b805af003246\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l"
Oct 30 22:06:57.306671 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.306344    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mn4fl\" (UniqueName: \"kubernetes.io/projected/0e128ead-13e5-41f3-9ae7-6f944b5f18ee-kube-api-access-mn4fl\") pod \"aws-ebs-csi-driver-node-bbkdz\" (UID: \"0e128ead-13e5-41f3-9ae7-6f944b5f18ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bbkdz"
Oct 30 22:06:57.306671 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.306377    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/87f77489-c825-4f2c-a6ce-70e06f68f187-tmp\") pod \"tuned-dr9c2\" (UID: \"87f77489-c825-4f2c-a6ce-70e06f68f187\") " pod="openshift-cluster-node-tuning-operator/tuned-dr9c2"
Oct 30 22:06:57.306671 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.306418    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2e4eaf5f-8a83-4681-b040-65676affdf7e-iptables-alerter-script\") pod \"iptables-alerter-vw5fg\" (UID: \"2e4eaf5f-8a83-4681-b040-65676affdf7e\") " pod="openshift-network-operator/iptables-alerter-vw5fg"
Oct 30 22:06:57.306671 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.306483    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b529dc48-9c57-4a0e-a4c0-49f53556ae34-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-69blh\" (UID: \"b529dc48-9c57-4a0e-a4c0-49f53556ae34\") " pod="openshift-insights/insights-runtime-extractor-69blh"
Oct 30 22:06:57.306671 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.306497    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/87f77489-c825-4f2c-a6ce-70e06f68f187-tmp\") pod \"tuned-dr9c2\" (UID: \"87f77489-c825-4f2c-a6ce-70e06f68f187\") " pod="openshift-cluster-node-tuning-operator/tuned-dr9c2"
Oct 30 22:06:57.306671 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.306511    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fngn\" (UniqueName: \"kubernetes.io/projected/98075172-cd56-41f2-8203-fc2639e95df9-kube-api-access-8fngn\") pod \"ingress-canary-wp8vp\" (UID: \"98075172-cd56-41f2-8203-fc2639e95df9\") " pod="openshift-ingress-canary/ingress-canary-wp8vp"
Oct 30 22:06:57.306671 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.306534    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b8503b30-31d6-4454-a3fd-8bea4a527b04-original-pull-secret\") pod \"global-pull-secret-syncer-rjtc8\" (UID: \"b8503b30-31d6-4454-a3fd-8bea4a527b04\") " pod="kube-system/global-pull-secret-syncer-rjtc8"
Oct 30 22:06:57.306671 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.306624    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/015f218d-bf1f-4e7e-bb5d-b805af003246-host-run-netns\") pod \"ovnkube-node-jcf8l\" (UID: \"015f218d-bf1f-4e7e-bb5d-b805af003246\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l"
Oct 30 22:06:57.306671 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.306644    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/015f218d-bf1f-4e7e-bb5d-b805af003246-host-run-ovn-kubernetes\") pod \"ovnkube-node-jcf8l\" (UID: \"015f218d-bf1f-4e7e-bb5d-b805af003246\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l"
Oct 30 22:06:57.306671 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.306660    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/87f77489-c825-4f2c-a6ce-70e06f68f187-host\") pod \"tuned-dr9c2\" (UID: \"87f77489-c825-4f2c-a6ce-70e06f68f187\") " pod="openshift-cluster-node-tuning-operator/tuned-dr9c2"
Oct 30 22:06:57.307153 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.306683    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7b50e755-6b0b-4d45-b7d3-c822127e6cb1-multus-daemon-config\") pod \"multus-4s4vc\" (UID: \"7b50e755-6b0b-4d45-b7d3-c822127e6cb1\") " pod="openshift-multus/multus-4s4vc"
Oct 30 22:06:57.307153 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.306716    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5d9bb\" (UniqueName: \"kubernetes.io/projected/7b50e755-6b0b-4d45-b7d3-c822127e6cb1-kube-api-access-5d9bb\") pod \"multus-4s4vc\" (UID: \"7b50e755-6b0b-4d45-b7d3-c822127e6cb1\") " pod="openshift-multus/multus-4s4vc"
Oct 30 22:06:57.307153 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.306866    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/15e3a459-855f-4181-825c-345e9b2e62e1-system-cni-dir\") pod \"multus-additional-cni-plugins-22c27\" (UID: \"15e3a459-855f-4181-825c-345e9b2e62e1\") " pod="openshift-multus/multus-additional-cni-plugins-22c27"
Oct 30 22:06:57.307153 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.306871    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b8503b30-31d6-4454-a3fd-8bea4a527b04-original-pull-secret\") pod \"global-pull-secret-syncer-rjtc8\" (UID: \"b8503b30-31d6-4454-a3fd-8bea4a527b04\") " pod="kube-system/global-pull-secret-syncer-rjtc8"
Oct 30 22:06:57.307153 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.306903    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ea63ef4-6b84-46e6-a753-bca60280734a-tmp-dir\") pod \"dns-default-c749j\" (UID: \"9ea63ef4-6b84-46e6-a753-bca60280734a\") " pod="openshift-dns/dns-default-c749j"
Oct 30 22:06:57.307153 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.306942    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfxls\" (UniqueName: \"kubernetes.io/projected/2e4eaf5f-8a83-4681-b040-65676affdf7e-kube-api-access-tfxls\") pod \"iptables-alerter-vw5fg\" (UID: \"2e4eaf5f-8a83-4681-b040-65676affdf7e\") " pod="openshift-network-operator/iptables-alerter-vw5fg"
Oct 30 22:06:57.307153 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.306970    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/015f218d-bf1f-4e7e-bb5d-b805af003246-host-kubelet\") pod \"ovnkube-node-jcf8l\" (UID: \"015f218d-bf1f-4e7e-bb5d-b805af003246\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l"
Oct 30 22:06:57.307153 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.306974    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9ea63ef4-6b84-46e6-a753-bca60280734a-tmp-dir\") pod \"dns-default-c749j\" (UID: \"9ea63ef4-6b84-46e6-a753-bca60280734a\") " pod="openshift-dns/dns-default-c749j"
Oct 30 22:06:57.307153 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.306996    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/015f218d-bf1f-4e7e-bb5d-b805af003246-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jcf8l\" (UID: \"015f218d-bf1f-4e7e-bb5d-b805af003246\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l"
Oct 30 22:06:57.307153 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.307000    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7b50e755-6b0b-4d45-b7d3-c822127e6cb1-multus-daemon-config\") pod \"multus-4s4vc\" (UID: \"7b50e755-6b0b-4d45-b7d3-c822127e6cb1\") " pod="openshift-multus/multus-4s4vc"
Oct 30 22:06:57.307153 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.307034    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9ea63ef4-6b84-46e6-a753-bca60280734a-metrics-tls\") pod \"dns-default-c749j\" (UID: \"9ea63ef4-6b84-46e6-a753-bca60280734a\") " pod="openshift-dns/dns-default-c749j"
Oct 30 22:06:57.307153 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.307101    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0e128ead-13e5-41f3-9ae7-6f944b5f18ee-device-dir\") pod \"aws-ebs-csi-driver-node-bbkdz\" (UID: \"0e128ead-13e5-41f3-9ae7-6f944b5f18ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bbkdz"
Oct 30 22:06:57.307153 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.307134    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7b50e755-6b0b-4d45-b7d3-c822127e6cb1-system-cni-dir\") pod \"multus-4s4vc\" (UID: \"7b50e755-6b0b-4d45-b7d3-c822127e6cb1\") " pod="openshift-multus/multus-4s4vc"
Oct 30 22:06:57.307153 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.307159    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7b50e755-6b0b-4d45-b7d3-c822127e6cb1-host-run-multus-certs\") pod \"multus-4s4vc\" (UID: \"7b50e755-6b0b-4d45-b7d3-c822127e6cb1\") " pod="openshift-multus/multus-4s4vc"
Oct 30 22:06:57.307715 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.307192    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e9c20fdf-bea5-4d6c-b252-b29b08bfdf63-serviceca\") pod \"node-ca-b74lk\" (UID: \"e9c20fdf-bea5-4d6c-b252-b29b08bfdf63\") " pod="openshift-image-registry/node-ca-b74lk"
Oct 30 22:06:57.307715 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.307250    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/015f218d-bf1f-4e7e-bb5d-b805af003246-systemd-units\") pod \"ovnkube-node-jcf8l\" (UID: \"015f218d-bf1f-4e7e-bb5d-b805af003246\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l"
Oct 30 22:06:57.307715 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.307268    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/87f77489-c825-4f2c-a6ce-70e06f68f187-etc-kubernetes\") pod \"tuned-dr9c2\" (UID: \"87f77489-c825-4f2c-a6ce-70e06f68f187\") " pod="openshift-cluster-node-tuning-operator/tuned-dr9c2"
Oct 30 22:06:57.307715 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.307296    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/87f77489-c825-4f2c-a6ce-70e06f68f187-etc-sysctl-conf\") pod \"tuned-dr9c2\" (UID: \"87f77489-c825-4f2c-a6ce-70e06f68f187\") " pod="openshift-cluster-node-tuning-operator/tuned-dr9c2"
Oct 30 22:06:57.307715 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.307312    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7b50e755-6b0b-4d45-b7d3-c822127e6cb1-host-run-k8s-cni-cncf-io\") pod \"multus-4s4vc\" (UID: \"7b50e755-6b0b-4d45-b7d3-c822127e6cb1\") " pod="openshift-multus/multus-4s4vc"
Oct 30 22:06:57.307715 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.307365    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7b50e755-6b0b-4d45-b7d3-c822127e6cb1-host-var-lib-kubelet\") pod \"multus-4s4vc\" (UID: \"7b50e755-6b0b-4d45-b7d3-c822127e6cb1\") " pod="openshift-multus/multus-4s4vc"
Oct 30 22:06:57.307715 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.307402    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7b50e755-6b0b-4d45-b7d3-c822127e6cb1-hostroot\") pod \"multus-4s4vc\" (UID: \"7b50e755-6b0b-4d45-b7d3-c822127e6cb1\") " pod="openshift-multus/multus-4s4vc"
Oct 30 22:06:57.307715 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.307434    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b529dc48-9c57-4a0e-a4c0-49f53556ae34-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-69blh\" (UID: \"b529dc48-9c57-4a0e-a4c0-49f53556ae34\") " pod="openshift-insights/insights-runtime-extractor-69blh"
Oct 30 22:06:57.307715 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.307450    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e9c20fdf-bea5-4d6c-b252-b29b08bfdf63-serviceca\") pod \"node-ca-b74lk\" (UID: \"e9c20fdf-bea5-4d6c-b252-b29b08bfdf63\") " pod="openshift-image-registry/node-ca-b74lk"
Oct 30 22:06:57.307715 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.307463    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cdcp\" (UniqueName: \"kubernetes.io/projected/e9c20fdf-bea5-4d6c-b252-b29b08bfdf63-kube-api-access-5cdcp\") pod \"node-ca-b74lk\" (UID: \"e9c20fdf-bea5-4d6c-b252-b29b08bfdf63\") " pod="openshift-image-registry/node-ca-b74lk"
Oct 30 22:06:57.307715 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.307490    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/015f218d-bf1f-4e7e-bb5d-b805af003246-run-openvswitch\") pod \"ovnkube-node-jcf8l\" (UID: \"015f218d-bf1f-4e7e-bb5d-b805af003246\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l"
Oct 30 22:06:57.307715 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.307507    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8286n\" (UniqueName: \"kubernetes.io/projected/dbf6b14c-5576-4ae3-bbb3-b3201aa7cb7c-kube-api-access-8286n\") pod \"network-check-target-vs5pt\" (UID: \"dbf6b14c-5576-4ae3-bbb3-b3201aa7cb7c\") " pod="openshift-network-diagnostics/network-check-target-vs5pt"
Oct 30 22:06:57.307715 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.307528    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/87f77489-c825-4f2c-a6ce-70e06f68f187-lib-modules\") pod \"tuned-dr9c2\" (UID: \"87f77489-c825-4f2c-a6ce-70e06f68f187\") " pod="openshift-cluster-node-tuning-operator/tuned-dr9c2"
Oct 30 22:06:57.307715 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.307557    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7b50e755-6b0b-4d45-b7d3-c822127e6cb1-cni-binary-copy\") pod \"multus-4s4vc\" (UID: \"7b50e755-6b0b-4d45-b7d3-c822127e6cb1\") " pod="openshift-multus/multus-4s4vc"
Oct 30 22:06:57.307715 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.307585    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/15e3a459-855f-4181-825c-345e9b2e62e1-cnibin\") pod \"multus-additional-cni-plugins-22c27\" (UID: \"15e3a459-855f-4181-825c-345e9b2e62e1\") " pod="openshift-multus/multus-additional-cni-plugins-22c27"
Oct 30 22:06:57.307715 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.307712    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/015f218d-bf1f-4e7e-bb5d-b805af003246-host-cni-netd\") pod \"ovnkube-node-jcf8l\" (UID: \"015f218d-bf1f-4e7e-bb5d-b805af003246\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l"
Oct 30 22:06:57.308416 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.307731    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0e128ead-13e5-41f3-9ae7-6f944b5f18ee-registration-dir\") pod \"aws-ebs-csi-driver-node-bbkdz\" (UID: \"0e128ead-13e5-41f3-9ae7-6f944b5f18ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bbkdz"
Oct 30 22:06:57.308416 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.307746    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7b50e755-6b0b-4d45-b7d3-c822127e6cb1-cnibin\") pod \"multus-4s4vc\" (UID: \"7b50e755-6b0b-4d45-b7d3-c822127e6cb1\") " pod="openshift-multus/multus-4s4vc"
Oct 30 22:06:57.308416 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.307786    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7b50e755-6b0b-4d45-b7d3-c822127e6cb1-cni-binary-copy\") pod \"multus-4s4vc\" (UID: \"7b50e755-6b0b-4d45-b7d3-c822127e6cb1\") " pod="openshift-multus/multus-4s4vc"
Oct 30 22:06:57.308416 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.307800    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/15e3a459-855f-4181-825c-345e9b2e62e1-cni-binary-copy\") pod \"multus-additional-cni-plugins-22c27\" (UID: \"15e3a459-855f-4181-825c-345e9b2e62e1\") " pod="openshift-multus/multus-additional-cni-plugins-22c27"
Oct 30 22:06:57.308416 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.307830    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/15e3a459-855f-4181-825c-345e9b2e62e1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-22c27\" (UID: \"15e3a459-855f-4181-825c-345e9b2e62e1\") " pod="openshift-multus/multus-additional-cni-plugins-22c27"
Oct 30 22:06:57.308416 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.307871    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2p2x4\" (UniqueName: \"kubernetes.io/projected/b529dc48-9c57-4a0e-a4c0-49f53556ae34-kube-api-access-2p2x4\") pod \"insights-runtime-extractor-69blh\" (UID: \"b529dc48-9c57-4a0e-a4c0-49f53556ae34\") " pod="openshift-insights/insights-runtime-extractor-69blh"
Oct 30 22:06:57.308416 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.307954    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/015f218d-bf1f-4e7e-bb5d-b805af003246-run-systemd\") pod \"ovnkube-node-jcf8l\" (UID: \"015f218d-bf1f-4e7e-bb5d-b805af003246\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l"
Oct 30 22:06:57.308416 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.307994    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/015f218d-bf1f-4e7e-bb5d-b805af003246-var-lib-openvswitch\") pod \"ovnkube-node-jcf8l\" (UID: \"015f218d-bf1f-4e7e-bb5d-b805af003246\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l"
Oct 30 22:06:57.308416 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.308030    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/015f218d-bf1f-4e7e-bb5d-b805af003246-ovnkube-config\") pod \"ovnkube-node-jcf8l\" (UID: \"015f218d-bf1f-4e7e-bb5d-b805af003246\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l"
Oct 30 22:06:57.308416 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.308079    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/87f77489-c825-4f2c-a6ce-70e06f68f187-var-lib-kubelet\") pod \"tuned-dr9c2\" (UID: \"87f77489-c825-4f2c-a6ce-70e06f68f187\") " pod="openshift-cluster-node-tuning-operator/tuned-dr9c2"
Oct 30 22:06:57.308416 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.308120    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkshh\" (UniqueName: \"kubernetes.io/projected/da9b75bf-8af2-450f-9a98-c265eea48282-kube-api-access-mkshh\") pod \"node-resolver-4knxq\" (UID: \"da9b75bf-8af2-450f-9a98-c265eea48282\") " pod="openshift-dns/node-resolver-4knxq"
Oct 30 22:06:57.308416 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.308162    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a3d96078-3ec8-40aa-855a-d1ba4acf2e74-agent-certs\") pod \"konnectivity-agent-mm655\" (UID: \"a3d96078-3ec8-40aa-855a-d1ba4acf2e74\") " pod="kube-system/konnectivity-agent-mm655"
Oct 30 22:06:57.308416 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.308185    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7b50e755-6b0b-4d45-b7d3-c822127e6cb1-multus-cni-dir\") pod \"multus-4s4vc\" (UID: \"7b50e755-6b0b-4d45-b7d3-c822127e6cb1\") " pod="openshift-multus/multus-4s4vc"
Oct 30 22:06:57.308416 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.308203    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da9b75bf-8af2-450f-9a98-c265eea48282-tmp-dir\") pod \"node-resolver-4knxq\" (UID: \"da9b75bf-8af2-450f-9a98-c265eea48282\") " pod="openshift-dns/node-resolver-4knxq"
Oct 30 22:06:57.308416 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.308214    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/15e3a459-855f-4181-825c-345e9b2e62e1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-22c27\" (UID: \"15e3a459-855f-4181-825c-345e9b2e62e1\") " pod="openshift-multus/multus-additional-cni-plugins-22c27"
Oct 30 22:06:57.308416 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.308234    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ncxp\" (UniqueName: \"kubernetes.io/projected/15e3a459-855f-4181-825c-345e9b2e62e1-kube-api-access-4ncxp\") pod \"multus-additional-cni-plugins-22c27\" (UID: \"15e3a459-855f-4181-825c-345e9b2e62e1\") " pod="openshift-multus/multus-additional-cni-plugins-22c27"
Oct 30 22:06:57.308416 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.308262    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b529dc48-9c57-4a0e-a4c0-49f53556ae34-data-volume\") pod \"insights-runtime-extractor-69blh\" (UID: \"b529dc48-9c57-4a0e-a4c0-49f53556ae34\") " pod="openshift-insights/insights-runtime-extractor-69blh"
Oct 30 22:06:57.308416 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.308280    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b8503b30-31d6-4454-a3fd-8bea4a527b04-dbus\") pod \"global-pull-secret-syncer-rjtc8\" (UID: \"b8503b30-31d6-4454-a3fd-8bea4a527b04\") " pod="kube-system/global-pull-secret-syncer-rjtc8"
Oct 30 22:06:57.308416 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.308292    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/15e3a459-855f-4181-825c-345e9b2e62e1-cni-binary-copy\") pod \"multus-additional-cni-plugins-22c27\" (UID: \"15e3a459-855f-4181-825c-345e9b2e62e1\") " pod="openshift-multus/multus-additional-cni-plugins-22c27"
Oct 30 22:06:57.308416 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.308300    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da9b75bf-8af2-450f-9a98-c265eea48282-tmp-dir\") pod \"node-resolver-4knxq\" (UID: \"da9b75bf-8af2-450f-9a98-c265eea48282\") " pod="openshift-dns/node-resolver-4knxq"
Oct 30 22:06:57.308416 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.308291    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/015f218d-bf1f-4e7e-bb5d-b805af003246-ovnkube-config\") pod \"ovnkube-node-jcf8l\" (UID: \"015f218d-bf1f-4e7e-bb5d-b805af003246\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l"
Oct 30 22:06:57.308416 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.308414    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7b50e755-6b0b-4d45-b7d3-c822127e6cb1-multus-conf-dir\") pod \"multus-4s4vc\" (UID: \"7b50e755-6b0b-4d45-b7d3-c822127e6cb1\") " pod="openshift-multus/multus-4s4vc"
Oct 30 22:06:57.309276 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.308456    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b529dc48-9c57-4a0e-a4c0-49f53556ae34-data-volume\") pod \"insights-runtime-extractor-69blh\" (UID: \"b529dc48-9c57-4a0e-a4c0-49f53556ae34\") " pod="openshift-insights/insights-runtime-extractor-69blh"
Oct 30 22:06:57.309276 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.308485    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckg9z\" (UniqueName: \"kubernetes.io/projected/9ea63ef4-6b84-46e6-a753-bca60280734a-kube-api-access-ckg9z\") pod \"dns-default-c749j\" (UID: \"9ea63ef4-6b84-46e6-a753-bca60280734a\") " pod="openshift-dns/dns-default-c749j"
Oct 30 22:06:57.309276 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.308522    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b529dc48-9c57-4a0e-a4c0-49f53556ae34-crio-socket\") pod \"insights-runtime-extractor-69blh\" (UID: \"b529dc48-9c57-4a0e-a4c0-49f53556ae34\") " pod="openshift-insights/insights-runtime-extractor-69blh"
Oct 30 22:06:57.309276 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.308545    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e9c20fdf-bea5-4d6c-b252-b29b08bfdf63-host\") pod \"node-ca-b74lk\" (UID: \"e9c20fdf-bea5-4d6c-b252-b29b08bfdf63\") " pod="openshift-image-registry/node-ca-b74lk"
Oct 30 22:06:57.309276 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.308562    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a3d96078-3ec8-40aa-855a-d1ba4acf2e74-agent-certs\") pod \"konnectivity-agent-mm655\" (UID: \"a3d96078-3ec8-40aa-855a-d1ba4acf2e74\") " pod="kube-system/konnectivity-agent-mm655"
Oct 30 22:06:57.309276 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.308584    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/015f218d-bf1f-4e7e-bb5d-b805af003246-host-slash\") pod \"ovnkube-node-jcf8l\" (UID: \"015f218d-bf1f-4e7e-bb5d-b805af003246\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l"
Oct 30 22:06:57.309276 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.308636    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/015f218d-bf1f-4e7e-bb5d-b805af003246-log-socket\") pod \"ovnkube-node-jcf8l\" (UID: \"015f218d-bf1f-4e7e-bb5d-b805af003246\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l"
Oct 30 22:06:57.309276 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.308678    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/015f218d-bf1f-4e7e-bb5d-b805af003246-env-overrides\") pod \"ovnkube-node-jcf8l\" (UID: \"015f218d-bf1f-4e7e-bb5d-b805af003246\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l"
Oct 30 22:06:57.309276 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.308708    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57q7f\" (UniqueName: \"kubernetes.io/projected/015f218d-bf1f-4e7e-bb5d-b805af003246-kube-api-access-57q7f\") pod \"ovnkube-node-jcf8l\" (UID: \"015f218d-bf1f-4e7e-bb5d-b805af003246\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l"
Oct 30 22:06:57.309276 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.308844    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/15e3a459-855f-4181-825c-345e9b2e62e1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-22c27\" (UID: \"15e3a459-855f-4181-825c-345e9b2e62e1\") " pod="openshift-multus/multus-additional-cni-plugins-22c27"
Oct 30 22:06:57.309276 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.308876    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/015f218d-bf1f-4e7e-bb5d-b805af003246-run-ovn\") pod \"ovnkube-node-jcf8l\" (UID: \"015f218d-bf1f-4e7e-bb5d-b805af003246\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l"
Oct 30 22:06:57.309276 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.308928    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/015f218d-bf1f-4e7e-bb5d-b805af003246-ovn-node-metrics-cert\") pod \"ovnkube-node-jcf8l\" (UID: \"015f218d-bf1f-4e7e-bb5d-b805af003246\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l"
Oct 30 22:06:57.309276 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.308929    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/015f218d-bf1f-4e7e-bb5d-b805af003246-env-overrides\") pod \"ovnkube-node-jcf8l\" (UID: \"015f218d-bf1f-4e7e-bb5d-b805af003246\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l"
Oct 30 22:06:57.309276 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.309075    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0e128ead-13e5-41f3-9ae7-6f944b5f18ee-socket-dir\") pod \"aws-ebs-csi-driver-node-bbkdz\" (UID: \"0e128ead-13e5-41f3-9ae7-6f944b5f18ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bbkdz"
Oct 30 22:06:57.309276 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.309099    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/015f218d-bf1f-4e7e-bb5d-b805af003246-ovn-node-metrics-cert\") pod \"ovnkube-node-jcf8l\" (UID: \"015f218d-bf1f-4e7e-bb5d-b805af003246\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l"
Oct 30 22:06:57.309276 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.309107    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/15e3a459-855f-4181-825c-345e9b2e62e1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-22c27\" (UID: \"15e3a459-855f-4181-825c-345e9b2e62e1\") " pod="openshift-multus/multus-additional-cni-plugins-22c27"
Oct 30 22:06:57.309276 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.309105    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/87f77489-c825-4f2c-a6ce-70e06f68f187-etc-systemd\") pod \"tuned-dr9c2\" (UID: \"87f77489-c825-4f2c-a6ce-70e06f68f187\") " pod="openshift-cluster-node-tuning-operator/tuned-dr9c2"
Oct 30 22:06:57.319135 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.319117    4477 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4knxq_da9b75bf-8af2-450f-9a98-c265eea48282/dns-node-resolver/0.log"
Oct 30 22:06:57.321980 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.321950    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frxsh\" (UniqueName: \"kubernetes.io/projected/fe4c45fd-690b-4111-9d15-93dc183b3e46-kube-api-access-frxsh\") pod \"network-metrics-daemon-6hhw9\" (UID: \"fe4c45fd-690b-4111-9d15-93dc183b3e46\") " pod="openshift-multus/network-metrics-daemon-6hhw9"
Oct 30 22:06:57.322109 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.321978    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5d6l6\" (UniqueName: \"kubernetes.io/projected/d4e7cd84-8a3f-4ef7-970c-008a9c895a47-kube-api-access-5d6l6\") pod \"migrator-66df5d496b-nzx7v\" (UID: \"d4e7cd84-8a3f-4ef7-970c-008a9c895a47\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-nzx7v"
Oct 30 22:06:57.322194 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.322175    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hc5rj\" (UniqueName: \"kubernetes.io/projected/87f77489-c825-4f2c-a6ce-70e06f68f187-kube-api-access-hc5rj\") pod \"tuned-dr9c2\" (UID: \"87f77489-c825-4f2c-a6ce-70e06f68f187\") " pod="openshift-cluster-node-tuning-operator/tuned-dr9c2"
Oct 30 22:06:57.322599 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.322584    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fngn\" (UniqueName: \"kubernetes.io/projected/98075172-cd56-41f2-8203-fc2639e95df9-kube-api-access-8fngn\") pod \"ingress-canary-wp8vp\" (UID: \"98075172-cd56-41f2-8203-fc2639e95df9\") " pod="openshift-ingress-canary/ingress-canary-wp8vp"
Oct 30 22:06:57.323797 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.323781    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfxls\" (UniqueName: \"kubernetes.io/projected/2e4eaf5f-8a83-4681-b040-65676affdf7e-kube-api-access-tfxls\") pod \"iptables-alerter-vw5fg\" (UID: \"2e4eaf5f-8a83-4681-b040-65676affdf7e\") " pod="openshift-network-operator/iptables-alerter-vw5fg"
Oct 30 22:06:57.324275 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.324258    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5d9bb\" (UniqueName: \"kubernetes.io/projected/7b50e755-6b0b-4d45-b7d3-c822127e6cb1-kube-api-access-5d9bb\") pod \"multus-4s4vc\" (UID: \"7b50e755-6b0b-4d45-b7d3-c822127e6cb1\") " pod="openshift-multus/multus-4s4vc"
Oct 30 22:06:57.324460 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.324444    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mn4fl\" (UniqueName: \"kubernetes.io/projected/0e128ead-13e5-41f3-9ae7-6f944b5f18ee-kube-api-access-mn4fl\") pod \"aws-ebs-csi-driver-node-bbkdz\" (UID: \"0e128ead-13e5-41f3-9ae7-6f944b5f18ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bbkdz"
Oct 30 22:06:57.324505 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.324483    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cdcp\" (UniqueName: \"kubernetes.io/projected/e9c20fdf-bea5-4d6c-b252-b29b08bfdf63-kube-api-access-5cdcp\") pod \"node-ca-b74lk\" (UID: \"e9c20fdf-bea5-4d6c-b252-b29b08bfdf63\") " pod="openshift-image-registry/node-ca-b74lk"
Oct 30 22:06:57.328156 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.328138    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2p2x4\" (UniqueName: \"kubernetes.io/projected/b529dc48-9c57-4a0e-a4c0-49f53556ae34-kube-api-access-2p2x4\") pod \"insights-runtime-extractor-69blh\" (UID: \"b529dc48-9c57-4a0e-a4c0-49f53556ae34\") " pod="openshift-insights/insights-runtime-extractor-69blh"
Oct 30 22:06:57.328224 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.328208    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkshh\" (UniqueName: \"kubernetes.io/projected/da9b75bf-8af2-450f-9a98-c265eea48282-kube-api-access-mkshh\") pod \"node-resolver-4knxq\" (UID: \"da9b75bf-8af2-450f-9a98-c265eea48282\") " pod="openshift-dns/node-resolver-4knxq"
Oct 30 22:06:57.328518 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.328503    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8286n\" (UniqueName: \"kubernetes.io/projected/dbf6b14c-5576-4ae3-bbb3-b3201aa7cb7c-kube-api-access-8286n\") pod \"network-check-target-vs5pt\" (UID: \"dbf6b14c-5576-4ae3-bbb3-b3201aa7cb7c\") " pod="openshift-network-diagnostics/network-check-target-vs5pt"
Oct 30 22:06:57.328655 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.328640    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckg9z\" (UniqueName: \"kubernetes.io/projected/9ea63ef4-6b84-46e6-a753-bca60280734a-kube-api-access-ckg9z\") pod \"dns-default-c749j\" (UID: \"9ea63ef4-6b84-46e6-a753-bca60280734a\") " pod="openshift-dns/dns-default-c749j"
Oct 30 22:06:57.328785 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.328769    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ncxp\" (UniqueName: \"kubernetes.io/projected/15e3a459-855f-4181-825c-345e9b2e62e1-kube-api-access-4ncxp\") pod \"multus-additional-cni-plugins-22c27\" (UID: \"15e3a459-855f-4181-825c-345e9b2e62e1\") " pod="openshift-multus/multus-additional-cni-plugins-22c27"
Oct 30 22:06:57.329281 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.329261    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57q7f\" (UniqueName: \"kubernetes.io/projected/015f218d-bf1f-4e7e-bb5d-b805af003246-kube-api-access-57q7f\") pod \"ovnkube-node-jcf8l\" (UID: \"015f218d-bf1f-4e7e-bb5d-b805af003246\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l"
Oct 30 22:06:57.409653 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.409598    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7b50e755-6b0b-4d45-b7d3-c822127e6cb1-multus-cni-dir\") pod \"multus-4s4vc\" (UID: \"7b50e755-6b0b-4d45-b7d3-c822127e6cb1\") " pod="openshift-multus/multus-4s4vc"
Oct 30 22:06:57.409653 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.409641    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b8503b30-31d6-4454-a3fd-8bea4a527b04-dbus\") pod \"global-pull-secret-syncer-rjtc8\" (UID: \"b8503b30-31d6-4454-a3fd-8bea4a527b04\") " pod="kube-system/global-pull-secret-syncer-rjtc8"
Oct 30 22:06:57.409762 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.409657    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7b50e755-6b0b-4d45-b7d3-c822127e6cb1-multus-conf-dir\") pod \"multus-4s4vc\" (UID: \"7b50e755-6b0b-4d45-b7d3-c822127e6cb1\") " pod="openshift-multus/multus-4s4vc"
Oct 30 22:06:57.409762 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.409677    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b529dc48-9c57-4a0e-a4c0-49f53556ae34-crio-socket\") pod \"insights-runtime-extractor-69blh\" (UID: \"b529dc48-9c57-4a0e-a4c0-49f53556ae34\") " pod="openshift-insights/insights-runtime-extractor-69blh"
Oct 30 22:06:57.409762 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.409700    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e9c20fdf-bea5-4d6c-b252-b29b08bfdf63-host\") pod \"node-ca-b74lk\" (UID: \"e9c20fdf-bea5-4d6c-b252-b29b08bfdf63\") " pod="openshift-image-registry/node-ca-b74lk"
Oct 30 22:06:57.409762 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.409725    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7b50e755-6b0b-4d45-b7d3-c822127e6cb1-multus-cni-dir\") pod \"multus-4s4vc\" (UID: \"7b50e755-6b0b-4d45-b7d3-c822127e6cb1\") " pod="openshift-multus/multus-4s4vc"
Oct 30 22:06:57.409762 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.409741    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e9c20fdf-bea5-4d6c-b252-b29b08bfdf63-host\") pod \"node-ca-b74lk\" (UID: \"e9c20fdf-bea5-4d6c-b252-b29b08bfdf63\") " pod="openshift-image-registry/node-ca-b74lk"
Oct 30 22:06:57.409762 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.409735    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b8503b30-31d6-4454-a3fd-8bea4a527b04-dbus\") pod \"global-pull-secret-syncer-rjtc8\" (UID: \"b8503b30-31d6-4454-a3fd-8bea4a527b04\") " pod="kube-system/global-pull-secret-syncer-rjtc8"
Oct 30 22:06:57.409762 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.409750    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b529dc48-9c57-4a0e-a4c0-49f53556ae34-crio-socket\") pod \"insights-runtime-extractor-69blh\" (UID: \"b529dc48-9c57-4a0e-a4c0-49f53556ae34\") " pod="openshift-insights/insights-runtime-extractor-69blh"
Oct 30 22:06:57.409977 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.409731    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7b50e755-6b0b-4d45-b7d3-c822127e6cb1-multus-conf-dir\") pod \"multus-4s4vc\" (UID: \"7b50e755-6b0b-4d45-b7d3-c822127e6cb1\") " pod="openshift-multus/multus-4s4vc"
Oct 30 22:06:57.409977 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.409786    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/015f218d-bf1f-4e7e-bb5d-b805af003246-host-slash\") pod \"ovnkube-node-jcf8l\" (UID: \"015f218d-bf1f-4e7e-bb5d-b805af003246\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l"
Oct 30 22:06:57.409977 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.409805    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/015f218d-bf1f-4e7e-bb5d-b805af003246-log-socket\") pod \"ovnkube-node-jcf8l\" (UID: \"015f218d-bf1f-4e7e-bb5d-b805af003246\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l"
Oct 30 22:06:57.409977 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.409822    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/015f218d-bf1f-4e7e-bb5d-b805af003246-host-slash\") pod \"ovnkube-node-jcf8l\" (UID: \"015f218d-bf1f-4e7e-bb5d-b805af003246\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l"
Oct 30 22:06:57.409977 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.409828    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/015f218d-bf1f-4e7e-bb5d-b805af003246-run-ovn\") pod \"ovnkube-node-jcf8l\" (UID: \"015f218d-bf1f-4e7e-bb5d-b805af003246\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l"
Oct 30 22:06:57.409977 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.409851    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/015f218d-bf1f-4e7e-bb5d-b805af003246-log-socket\") pod \"ovnkube-node-jcf8l\" (UID: \"015f218d-bf1f-4e7e-bb5d-b805af003246\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l"
Oct 30 22:06:57.409977 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.409860    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0e128ead-13e5-41f3-9ae7-6f944b5f18ee-socket-dir\") pod \"aws-ebs-csi-driver-node-bbkdz\" (UID: \"0e128ead-13e5-41f3-9ae7-6f944b5f18ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bbkdz"
Oct 30 22:06:57.409977 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.409863    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/015f218d-bf1f-4e7e-bb5d-b805af003246-run-ovn\") pod \"ovnkube-node-jcf8l\" (UID: \"015f218d-bf1f-4e7e-bb5d-b805af003246\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l"
Oct 30 22:06:57.409977 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.409894    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/87f77489-c825-4f2c-a6ce-70e06f68f187-etc-systemd\") pod \"tuned-dr9c2\" (UID: \"87f77489-c825-4f2c-a6ce-70e06f68f187\") " pod="openshift-cluster-node-tuning-operator/tuned-dr9c2"
Oct 30 22:06:57.409977 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.409909    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0e128ead-13e5-41f3-9ae7-6f944b5f18ee-socket-dir\") pod \"aws-ebs-csi-driver-node-bbkdz\" (UID: \"0e128ead-13e5-41f3-9ae7-6f944b5f18ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bbkdz"
Oct 30 22:06:57.409977 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.409910    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7b50e755-6b0b-4d45-b7d3-c822127e6cb1-multus-socket-dir-parent\") pod \"multus-4s4vc\" (UID: \"7b50e755-6b0b-4d45-b7d3-c822127e6cb1\") " pod="openshift-multus/multus-4s4vc"
Oct 30 22:06:57.409977 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.409950    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7b50e755-6b0b-4d45-b7d3-c822127e6cb1-multus-socket-dir-parent\") pod \"multus-4s4vc\" (UID: \"7b50e755-6b0b-4d45-b7d3-c822127e6cb1\") " pod="openshift-multus/multus-4s4vc"
Oct 30 22:06:57.409977 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.409954    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7b50e755-6b0b-4d45-b7d3-c822127e6cb1-host-run-netns\") pod \"multus-4s4vc\" (UID: \"7b50e755-6b0b-4d45-b7d3-c822127e6cb1\") " pod="openshift-multus/multus-4s4vc"
Oct 30 22:06:57.409977 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.409979    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/15e3a459-855f-4181-825c-345e9b2e62e1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-22c27\" (UID: \"15e3a459-855f-4181-825c-345e9b2e62e1\") " pod="openshift-multus/multus-additional-cni-plugins-22c27"
Oct 30 22:06:57.410419 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.409985    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7b50e755-6b0b-4d45-b7d3-c822127e6cb1-host-run-netns\") pod \"multus-4s4vc\" (UID: \"7b50e755-6b0b-4d45-b7d3-c822127e6cb1\") " pod="openshift-multus/multus-4s4vc"
Oct 30 22:06:57.410419 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.410003    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b8503b30-31d6-4454-a3fd-8bea4a527b04-kubelet-config\") pod \"global-pull-secret-syncer-rjtc8\" (UID: \"b8503b30-31d6-4454-a3fd-8bea4a527b04\") " pod="kube-system/global-pull-secret-syncer-rjtc8"
Oct 30 22:06:57.410419 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.409949    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/87f77489-c825-4f2c-a6ce-70e06f68f187-etc-systemd\") pod \"tuned-dr9c2\" (UID: \"87f77489-c825-4f2c-a6ce-70e06f68f187\") " pod="openshift-cluster-node-tuning-operator/tuned-dr9c2"
Oct 30 22:06:57.410419 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.410021    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/015f218d-bf1f-4e7e-bb5d-b805af003246-node-log\") pod \"ovnkube-node-jcf8l\" (UID: \"015f218d-bf1f-4e7e-bb5d-b805af003246\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l"
Oct 30 22:06:57.410419 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.410071    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/015f218d-bf1f-4e7e-bb5d-b805af003246-node-log\") pod \"ovnkube-node-jcf8l\" (UID: \"015f218d-bf1f-4e7e-bb5d-b805af003246\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l"
Oct 30 22:06:57.410419 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.410077    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b8503b30-31d6-4454-a3fd-8bea4a527b04-kubelet-config\") pod \"global-pull-secret-syncer-rjtc8\" (UID: \"b8503b30-31d6-4454-a3fd-8bea4a527b04\") " pod="kube-system/global-pull-secret-syncer-rjtc8"
Oct 30 22:06:57.410419 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.410095    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/87f77489-c825-4f2c-a6ce-70e06f68f187-etc-modprobe-d\") pod \"tuned-dr9c2\" (UID: \"87f77489-c825-4f2c-a6ce-70e06f68f187\") " pod="openshift-cluster-node-tuning-operator/tuned-dr9c2"
Oct 30 22:06:57.410419 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.410111    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/87f77489-c825-4f2c-a6ce-70e06f68f187-etc-sysconfig\") pod \"tuned-dr9c2\" (UID: \"87f77489-c825-4f2c-a6ce-70e06f68f187\") " pod="openshift-cluster-node-tuning-operator/tuned-dr9c2"
Oct 30 22:06:57.410419 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.410131    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/15e3a459-855f-4181-825c-345e9b2e62e1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-22c27\" (UID: \"15e3a459-855f-4181-825c-345e9b2e62e1\") " pod="openshift-multus/multus-additional-cni-plugins-22c27"
Oct 30 22:06:57.410419 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.410150    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7b50e755-6b0b-4d45-b7d3-c822127e6cb1-etc-kubernetes\") pod \"multus-4s4vc\" (UID: \"7b50e755-6b0b-4d45-b7d3-c822127e6cb1\") " pod="openshift-multus/multus-4s4vc"
Oct 30 22:06:57.410419 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.410152    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/87f77489-c825-4f2c-a6ce-70e06f68f187-etc-sysconfig\") pod \"tuned-dr9c2\" (UID: \"87f77489-c825-4f2c-a6ce-70e06f68f187\") " pod="openshift-cluster-node-tuning-operator/tuned-dr9c2"
Oct 30 22:06:57.410419 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.410164    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/015f218d-bf1f-4e7e-bb5d-b805af003246-etc-openvswitch\") pod \"ovnkube-node-jcf8l\" (UID: \"015f218d-bf1f-4e7e-bb5d-b805af003246\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l"
Oct 30 22:06:57.410419 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.410183    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7b50e755-6b0b-4d45-b7d3-c822127e6cb1-etc-kubernetes\") pod \"multus-4s4vc\" (UID: \"7b50e755-6b0b-4d45-b7d3-c822127e6cb1\") " pod="openshift-multus/multus-4s4vc"
Oct 30 22:06:57.410419 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.410187    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0e128ead-13e5-41f3-9ae7-6f944b5f18ee-etc-selinux\") pod \"aws-ebs-csi-driver-node-bbkdz\" (UID: \"0e128ead-13e5-41f3-9ae7-6f944b5f18ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bbkdz"
Oct 30 22:06:57.410419 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.410209    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/87f77489-c825-4f2c-a6ce-70e06f68f187-etc-modprobe-d\") pod \"tuned-dr9c2\" (UID: \"87f77489-c825-4f2c-a6ce-70e06f68f187\") " pod="openshift-cluster-node-tuning-operator/tuned-dr9c2"
Oct 30 22:06:57.410419 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.410214    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/87f77489-c825-4f2c-a6ce-70e06f68f187-run\") pod \"tuned-dr9c2\" (UID: \"87f77489-c825-4f2c-a6ce-70e06f68f187\") " pod="openshift-cluster-node-tuning-operator/tuned-dr9c2"
Oct 30 22:06:57.410419 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.410216    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/015f218d-bf1f-4e7e-bb5d-b805af003246-etc-openvswitch\") pod \"ovnkube-node-jcf8l\" (UID: \"015f218d-bf1f-4e7e-bb5d-b805af003246\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l"
Oct 30 22:06:57.410419 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.410226    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0e128ead-13e5-41f3-9ae7-6f944b5f18ee-etc-selinux\") pod \"aws-ebs-csi-driver-node-bbkdz\" (UID: \"0e128ead-13e5-41f3-9ae7-6f944b5f18ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bbkdz"
Oct 30 22:06:57.410419 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.410237    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7b50e755-6b0b-4d45-b7d3-c822127e6cb1-host-var-lib-cni-multus\") pod \"multus-4s4vc\" (UID: \"7b50e755-6b0b-4d45-b7d3-c822127e6cb1\") " pod="openshift-multus/multus-4s4vc"
Oct 30 22:06:57.410419 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.410251    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7b50e755-6b0b-4d45-b7d3-c822127e6cb1-host-var-lib-cni-bin\") pod \"multus-4s4vc\" (UID: \"7b50e755-6b0b-4d45-b7d3-c822127e6cb1\") " pod="openshift-multus/multus-4s4vc"
Oct 30 22:06:57.410419 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.410268    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0e128ead-13e5-41f3-9ae7-6f944b5f18ee-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bbkdz\" (UID: \"0e128ead-13e5-41f3-9ae7-6f944b5f18ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bbkdz"
Oct 30 22:06:57.410419 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.410269    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/87f77489-c825-4f2c-a6ce-70e06f68f187-run\") pod \"tuned-dr9c2\" (UID: \"87f77489-c825-4f2c-a6ce-70e06f68f187\") " pod="openshift-cluster-node-tuning-operator/tuned-dr9c2"
Oct 30 22:06:57.410419 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.410273    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7b50e755-6b0b-4d45-b7d3-c822127e6cb1-host-var-lib-cni-multus\") pod \"multus-4s4vc\" (UID: \"7b50e755-6b0b-4d45-b7d3-c822127e6cb1\") " pod="openshift-multus/multus-4s4vc"
Oct 30 22:06:57.410419 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.410284    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/87f77489-c825-4f2c-a6ce-70e06f68f187-etc-sysctl-d\") pod \"tuned-dr9c2\" (UID: \"87f77489-c825-4f2c-a6ce-70e06f68f187\") " pod="openshift-cluster-node-tuning-operator/tuned-dr9c2"
Oct 30 22:06:57.410419 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.410305    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/da9b75bf-8af2-450f-9a98-c265eea48282-hosts-file\") pod \"node-resolver-4knxq\" (UID: \"da9b75bf-8af2-450f-9a98-c265eea48282\") " pod="openshift-dns/node-resolver-4knxq"
Oct 30 22:06:57.410419 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.410314    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7b50e755-6b0b-4d45-b7d3-c822127e6cb1-host-var-lib-cni-bin\") pod \"multus-4s4vc\" (UID: \"7b50e755-6b0b-4d45-b7d3-c822127e6cb1\") " pod="openshift-multus/multus-4s4vc"
Oct 30 22:06:57.410419 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.410317    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0e128ead-13e5-41f3-9ae7-6f944b5f18ee-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bbkdz\" (UID: \"0e128ead-13e5-41f3-9ae7-6f944b5f18ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bbkdz"
Oct 30 22:06:57.410419 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.410323    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe4c45fd-690b-4111-9d15-93dc183b3e46-metrics-certs\") pod \"network-metrics-daemon-6hhw9\" (UID: \"fe4c45fd-690b-4111-9d15-93dc183b3e46\") " pod="openshift-multus/network-metrics-daemon-6hhw9"
Oct 30 22:06:57.410419 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.410353    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0e128ead-13e5-41f3-9ae7-6f944b5f18ee-sys-fs\") pod \"aws-ebs-csi-driver-node-bbkdz\" (UID: \"0e128ead-13e5-41f3-9ae7-6f944b5f18ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bbkdz"
Oct 30 22:06:57.410419 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.410358    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/da9b75bf-8af2-450f-9a98-c265eea48282-hosts-file\") pod \"node-resolver-4knxq\" (UID: \"da9b75bf-8af2-450f-9a98-c265eea48282\") " pod="openshift-dns/node-resolver-4knxq"
Oct 30 22:06:57.410419 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.410370    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/87f77489-c825-4f2c-a6ce-70e06f68f187-etc-sysctl-d\") pod \"tuned-dr9c2\" (UID: \"87f77489-c825-4f2c-a6ce-70e06f68f187\") " pod="openshift-cluster-node-tuning-operator/tuned-dr9c2"
Oct 30 22:06:57.410419 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.410376    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/87f77489-c825-4f2c-a6ce-70e06f68f187-sys\") pod \"tuned-dr9c2\" (UID: \"87f77489-c825-4f2c-a6ce-70e06f68f187\") " pod="openshift-cluster-node-tuning-operator/tuned-dr9c2"
Oct 30 22:06:57.410419 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.410400    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0e128ead-13e5-41f3-9ae7-6f944b5f18ee-sys-fs\") pod \"aws-ebs-csi-driver-node-bbkdz\" (UID: \"0e128ead-13e5-41f3-9ae7-6f944b5f18ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bbkdz"
Oct 30 22:06:57.410419 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.410418    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/87f77489-c825-4f2c-a6ce-70e06f68f187-sys\") pod \"tuned-dr9c2\" (UID: \"87f77489-c825-4f2c-a6ce-70e06f68f187\") " pod="openshift-cluster-node-tuning-operator/tuned-dr9c2"
Oct 30 22:06:57.410419 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:06:57.410425    4477 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 22:06:57.410419 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.410440    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2e4eaf5f-8a83-4681-b040-65676affdf7e-host-slash\") pod \"iptables-alerter-vw5fg\" (UID: \"2e4eaf5f-8a83-4681-b040-65676affdf7e\") " pod="openshift-network-operator/iptables-alerter-vw5fg"
Oct 30 22:06:57.411535 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.410419    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2e4eaf5f-8a83-4681-b040-65676affdf7e-host-slash\") pod \"iptables-alerter-vw5fg\" (UID: \"2e4eaf5f-8a83-4681-b040-65676affdf7e\") " pod="openshift-network-operator/iptables-alerter-vw5fg"
Oct 30 22:06:57.411535 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:06:57.410483    4477 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe4c45fd-690b-4111-9d15-93dc183b3e46-metrics-certs podName:fe4c45fd-690b-4111-9d15-93dc183b3e46 nodeName:}" failed. No retries permitted until 2026-10-30 22:06:57.910470019 +0000 UTC m=+1.836396232 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fe4c45fd-690b-4111-9d15-93dc183b3e46-metrics-certs") pod "network-metrics-daemon-6hhw9" (UID: "fe4c45fd-690b-4111-9d15-93dc183b3e46") : secret "metrics-daemon-secret" not found
Oct 30 22:06:57.411535 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.410503    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/015f218d-bf1f-4e7e-bb5d-b805af003246-host-cni-bin\") pod \"ovnkube-node-jcf8l\" (UID: \"015f218d-bf1f-4e7e-bb5d-b805af003246\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l"
Oct 30 22:06:57.411535 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.410519    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b529dc48-9c57-4a0e-a4c0-49f53556ae34-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-69blh\" (UID: \"b529dc48-9c57-4a0e-a4c0-49f53556ae34\") " pod="openshift-insights/insights-runtime-extractor-69blh"
Oct 30 22:06:57.411535 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.410534    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/015f218d-bf1f-4e7e-bb5d-b805af003246-host-run-netns\") pod \"ovnkube-node-jcf8l\" (UID: \"015f218d-bf1f-4e7e-bb5d-b805af003246\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l"
Oct 30 22:06:57.411535 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.410545    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/015f218d-bf1f-4e7e-bb5d-b805af003246-host-cni-bin\") pod \"ovnkube-node-jcf8l\" (UID: \"015f218d-bf1f-4e7e-bb5d-b805af003246\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l"
Oct 30 22:06:57.411535 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.410548    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/015f218d-bf1f-4e7e-bb5d-b805af003246-host-run-ovn-kubernetes\") pod \"ovnkube-node-jcf8l\" (UID: \"015f218d-bf1f-4e7e-bb5d-b805af003246\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l"
Oct 30 22:06:57.411535 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.410567    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/015f218d-bf1f-4e7e-bb5d-b805af003246-host-run-netns\") pod \"ovnkube-node-jcf8l\" (UID: \"015f218d-bf1f-4e7e-bb5d-b805af003246\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l"
Oct 30 22:06:57.411535 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.410567    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/87f77489-c825-4f2c-a6ce-70e06f68f187-host\") pod \"tuned-dr9c2\" (UID: \"87f77489-c825-4f2c-a6ce-70e06f68f187\") " pod="openshift-cluster-node-tuning-operator/tuned-dr9c2"
Oct 30 22:06:57.411535 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.410589    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/15e3a459-855f-4181-825c-345e9b2e62e1-system-cni-dir\") pod \"multus-additional-cni-plugins-22c27\" (UID: \"15e3a459-855f-4181-825c-345e9b2e62e1\") " pod="openshift-multus/multus-additional-cni-plugins-22c27"
Oct 30 22:06:57.411535 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.410593    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/87f77489-c825-4f2c-a6ce-70e06f68f187-host\") pod \"tuned-dr9c2\" (UID: \"87f77489-c825-4f2c-a6ce-70e06f68f187\") " pod="openshift-cluster-node-tuning-operator/tuned-dr9c2"
Oct 30 22:06:57.411535 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.410604    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/015f218d-bf1f-4e7e-bb5d-b805af003246-host-kubelet\") pod \"ovnkube-node-jcf8l\" (UID: \"015f218d-bf1f-4e7e-bb5d-b805af003246\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l"
Oct 30 22:06:57.411535 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.410632    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/15e3a459-855f-4181-825c-345e9b2e62e1-system-cni-dir\") pod \"multus-additional-cni-plugins-22c27\" (UID: \"15e3a459-855f-4181-825c-345e9b2e62e1\") " pod="openshift-multus/multus-additional-cni-plugins-22c27"
Oct 30 22:06:57.411535 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.410633    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/015f218d-bf1f-4e7e-bb5d-b805af003246-host-run-ovn-kubernetes\") pod \"ovnkube-node-jcf8l\" (UID: \"015f218d-bf1f-4e7e-bb5d-b805af003246\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l"
Oct 30 22:06:57.411535 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.410634    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/015f218d-bf1f-4e7e-bb5d-b805af003246-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jcf8l\" (UID: \"015f218d-bf1f-4e7e-bb5d-b805af003246\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l"
Oct 30 22:06:57.411535 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.410656    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/015f218d-bf1f-4e7e-bb5d-b805af003246-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jcf8l\" (UID: \"015f218d-bf1f-4e7e-bb5d-b805af003246\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l"
Oct 30 22:06:57.411535 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.410667    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9ea63ef4-6b84-46e6-a753-bca60280734a-metrics-tls\") pod \"dns-default-c749j\" (UID: \"9ea63ef4-6b84-46e6-a753-bca60280734a\") " pod="openshift-dns/dns-default-c749j"
Oct 30 22:06:57.411535 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.410669    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/015f218d-bf1f-4e7e-bb5d-b805af003246-host-kubelet\") pod \"ovnkube-node-jcf8l\" (UID: \"015f218d-bf1f-4e7e-bb5d-b805af003246\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l"
Oct 30 22:06:57.411535 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.410682    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0e128ead-13e5-41f3-9ae7-6f944b5f18ee-device-dir\") pod \"aws-ebs-csi-driver-node-bbkdz\" (UID: \"0e128ead-13e5-41f3-9ae7-6f944b5f18ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bbkdz"
Oct 30 22:06:57.411535 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.410697    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7b50e755-6b0b-4d45-b7d3-c822127e6cb1-system-cni-dir\") pod \"multus-4s4vc\" (UID: \"7b50e755-6b0b-4d45-b7d3-c822127e6cb1\") " pod="openshift-multus/multus-4s4vc"
Oct 30 22:06:57.411535 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.410717    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7b50e755-6b0b-4d45-b7d3-c822127e6cb1-host-run-multus-certs\") pod \"multus-4s4vc\" (UID: \"7b50e755-6b0b-4d45-b7d3-c822127e6cb1\") " pod="openshift-multus/multus-4s4vc"
Oct 30 22:06:57.411535 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.410742    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/015f218d-bf1f-4e7e-bb5d-b805af003246-systemd-units\") pod \"ovnkube-node-jcf8l\" (UID: \"015f218d-bf1f-4e7e-bb5d-b805af003246\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l"
Oct 30 22:06:57.411535 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.410742    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0e128ead-13e5-41f3-9ae7-6f944b5f18ee-device-dir\") pod \"aws-ebs-csi-driver-node-bbkdz\" (UID: \"0e128ead-13e5-41f3-9ae7-6f944b5f18ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bbkdz"
Oct 30 22:06:57.411535 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.410766    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/87f77489-c825-4f2c-a6ce-70e06f68f187-etc-kubernetes\") pod \"tuned-dr9c2\" (UID: \"87f77489-c825-4f2c-a6ce-70e06f68f187\") " pod="openshift-cluster-node-tuning-operator/tuned-dr9c2"
Oct 30 22:06:57.411535 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.410776    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7b50e755-6b0b-4d45-b7d3-c822127e6cb1-system-cni-dir\") pod \"multus-4s4vc\" (UID: \"7b50e755-6b0b-4d45-b7d3-c822127e6cb1\") " pod="openshift-multus/multus-4s4vc"
Oct 30 22:06:57.411535 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.410788    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/87f77489-c825-4f2c-a6ce-70e06f68f187-etc-sysctl-conf\") pod \"tuned-dr9c2\" (UID: \"87f77489-c825-4f2c-a6ce-70e06f68f187\") " pod="openshift-cluster-node-tuning-operator/tuned-dr9c2"
Oct 30 22:06:57.411535 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.410791    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/015f218d-bf1f-4e7e-bb5d-b805af003246-systemd-units\") pod \"ovnkube-node-jcf8l\" (UID: \"015f218d-bf1f-4e7e-bb5d-b805af003246\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l"
Oct 30 22:06:57.411535 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.410796    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7b50e755-6b0b-4d45-b7d3-c822127e6cb1-host-run-multus-certs\") pod \"multus-4s4vc\" (UID: \"7b50e755-6b0b-4d45-b7d3-c822127e6cb1\") " pod="openshift-multus/multus-4s4vc"
Oct 30 22:06:57.411535 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.410820    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/87f77489-c825-4f2c-a6ce-70e06f68f187-etc-kubernetes\") pod \"tuned-dr9c2\" (UID: \"87f77489-c825-4f2c-a6ce-70e06f68f187\") " pod="openshift-cluster-node-tuning-operator/tuned-dr9c2"
Oct 30 22:06:57.411535 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.410812    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7b50e755-6b0b-4d45-b7d3-c822127e6cb1-host-run-k8s-cni-cncf-io\") pod \"multus-4s4vc\" (UID: \"7b50e755-6b0b-4d45-b7d3-c822127e6cb1\") " pod="openshift-multus/multus-4s4vc"
Oct 30 22:06:57.411535 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.410846    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7b50e755-6b0b-4d45-b7d3-c822127e6cb1-host-run-k8s-cni-cncf-io\") pod \"multus-4s4vc\" (UID: \"7b50e755-6b0b-4d45-b7d3-c822127e6cb1\") " pod="openshift-multus/multus-4s4vc"
Oct 30 22:06:57.411535 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.410853    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7b50e755-6b0b-4d45-b7d3-c822127e6cb1-host-var-lib-kubelet\") pod \"multus-4s4vc\" (UID: \"7b50e755-6b0b-4d45-b7d3-c822127e6cb1\") " pod="openshift-multus/multus-4s4vc"
Oct 30 22:06:57.411535 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.410875    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7b50e755-6b0b-4d45-b7d3-c822127e6cb1-hostroot\") pod \"multus-4s4vc\" (UID: \"7b50e755-6b0b-4d45-b7d3-c822127e6cb1\") " pod="openshift-multus/multus-4s4vc"
Oct 30 22:06:57.411535 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.410899    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b529dc48-9c57-4a0e-a4c0-49f53556ae34-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-69blh\" (UID: \"b529dc48-9c57-4a0e-a4c0-49f53556ae34\") " pod="openshift-insights/insights-runtime-extractor-69blh"
Oct 30 22:06:57.411535 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.410922    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/015f218d-bf1f-4e7e-bb5d-b805af003246-run-openvswitch\") pod \"ovnkube-node-jcf8l\" (UID: \"015f218d-bf1f-4e7e-bb5d-b805af003246\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l"
Oct 30 22:06:57.411535 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.410926    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/87f77489-c825-4f2c-a6ce-70e06f68f187-etc-sysctl-conf\") pod \"tuned-dr9c2\" (UID: \"87f77489-c825-4f2c-a6ce-70e06f68f187\") " pod="openshift-cluster-node-tuning-operator/tuned-dr9c2"
Oct 30 22:06:57.411535 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.410949    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7b50e755-6b0b-4d45-b7d3-c822127e6cb1-hostroot\") pod \"multus-4s4vc\" (UID: \"7b50e755-6b0b-4d45-b7d3-c822127e6cb1\") " pod="openshift-multus/multus-4s4vc"
Oct 30 22:06:57.411535 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.410960    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/015f218d-bf1f-4e7e-bb5d-b805af003246-run-openvswitch\") pod \"ovnkube-node-jcf8l\" (UID: \"015f218d-bf1f-4e7e-bb5d-b805af003246\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l"
Oct 30 22:06:57.411535 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.410967    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7b50e755-6b0b-4d45-b7d3-c822127e6cb1-host-var-lib-kubelet\") pod \"multus-4s4vc\" (UID: \"7b50e755-6b0b-4d45-b7d3-c822127e6cb1\") " pod="openshift-multus/multus-4s4vc"
Oct 30 22:06:57.411535 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.410981    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/87f77489-c825-4f2c-a6ce-70e06f68f187-lib-modules\") pod \"tuned-dr9c2\" (UID: \"87f77489-c825-4f2c-a6ce-70e06f68f187\") " pod="openshift-cluster-node-tuning-operator/tuned-dr9c2"
Oct 30 22:06:57.411535 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.411014    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b529dc48-9c57-4a0e-a4c0-49f53556ae34-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-69blh\" (UID: \"b529dc48-9c57-4a0e-a4c0-49f53556ae34\") " pod="openshift-insights/insights-runtime-extractor-69blh"
Oct 30 22:06:57.411535 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:06:57.411000    4477 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 22:06:57.411535 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.411036    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/15e3a459-855f-4181-825c-345e9b2e62e1-cnibin\") pod \"multus-additional-cni-plugins-22c27\" (UID: \"15e3a459-855f-4181-825c-345e9b2e62e1\") " pod="openshift-multus/multus-additional-cni-plugins-22c27"
Oct 30 22:06:57.411535 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.411080    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/015f218d-bf1f-4e7e-bb5d-b805af003246-host-cni-netd\") pod \"ovnkube-node-jcf8l\" (UID: \"015f218d-bf1f-4e7e-bb5d-b805af003246\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l"
Oct 30 22:06:57.411535 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.411088    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/15e3a459-855f-4181-825c-345e9b2e62e1-cnibin\") pod \"multus-additional-cni-plugins-22c27\" (UID: \"15e3a459-855f-4181-825c-345e9b2e62e1\") " pod="openshift-multus/multus-additional-cni-plugins-22c27"
Oct 30 22:06:57.411535 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.410982    4477 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 22:06:57.411535 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.411111    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0e128ead-13e5-41f3-9ae7-6f944b5f18ee-registration-dir\") pod \"aws-ebs-csi-driver-node-bbkdz\" (UID: \"0e128ead-13e5-41f3-9ae7-6f944b5f18ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bbkdz"
Oct 30 22:06:57.411535 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.411144    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7b50e755-6b0b-4d45-b7d3-c822127e6cb1-cnibin\") pod \"multus-4s4vc\" (UID: \"7b50e755-6b0b-4d45-b7d3-c822127e6cb1\") " pod="openshift-multus/multus-4s4vc"
Oct 30 22:06:57.411535 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.411139    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/87f77489-c825-4f2c-a6ce-70e06f68f187-lib-modules\") pod \"tuned-dr9c2\" (UID: \"87f77489-c825-4f2c-a6ce-70e06f68f187\") " pod="openshift-cluster-node-tuning-operator/tuned-dr9c2"
Oct 30 22:06:57.411535 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.411151    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/015f218d-bf1f-4e7e-bb5d-b805af003246-host-cni-netd\") pod \"ovnkube-node-jcf8l\" (UID: \"015f218d-bf1f-4e7e-bb5d-b805af003246\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l"
Oct 30 22:06:57.411535 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:06:57.411167    4477 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b529dc48-9c57-4a0e-a4c0-49f53556ae34-insights-runtime-extractor-tls podName:b529dc48-9c57-4a0e-a4c0-49f53556ae34 nodeName:}" failed. No retries permitted until 2026-10-30 22:06:57.911150134 +0000 UTC m=+1.837076335 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b529dc48-9c57-4a0e-a4c0-49f53556ae34-insights-runtime-extractor-tls") pod "insights-runtime-extractor-69blh" (UID: "b529dc48-9c57-4a0e-a4c0-49f53556ae34") : secret "insights-runtime-extractor-tls" not found
Oct 30 22:06:57.411535 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.411164    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0e128ead-13e5-41f3-9ae7-6f944b5f18ee-registration-dir\") pod \"aws-ebs-csi-driver-node-bbkdz\" (UID: \"0e128ead-13e5-41f3-9ae7-6f944b5f18ee\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bbkdz"
Oct 30 22:06:57.411535 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.411186    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7b50e755-6b0b-4d45-b7d3-c822127e6cb1-cnibin\") pod \"multus-4s4vc\" (UID: \"7b50e755-6b0b-4d45-b7d3-c822127e6cb1\") " pod="openshift-multus/multus-4s4vc"
Oct 30 22:06:57.411535 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.411202    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/015f218d-bf1f-4e7e-bb5d-b805af003246-run-systemd\") pod \"ovnkube-node-jcf8l\" (UID: \"015f218d-bf1f-4e7e-bb5d-b805af003246\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l"
Oct 30 22:06:57.411535 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.411218    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/015f218d-bf1f-4e7e-bb5d-b805af003246-var-lib-openvswitch\") pod \"ovnkube-node-jcf8l\" (UID: \"015f218d-bf1f-4e7e-bb5d-b805af003246\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l"
Oct 30 22:06:57.411535 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.411237    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/87f77489-c825-4f2c-a6ce-70e06f68f187-var-lib-kubelet\") pod \"tuned-dr9c2\" (UID: \"87f77489-c825-4f2c-a6ce-70e06f68f187\") " pod="openshift-cluster-node-tuning-operator/tuned-dr9c2"
Oct 30 22:06:57.411535 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.411266    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/015f218d-bf1f-4e7e-bb5d-b805af003246-run-systemd\") pod \"ovnkube-node-jcf8l\" (UID: \"015f218d-bf1f-4e7e-bb5d-b805af003246\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l"
Oct 30 22:06:57.411535 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.411274    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/015f218d-bf1f-4e7e-bb5d-b805af003246-var-lib-openvswitch\") pod \"ovnkube-node-jcf8l\" (UID: \"015f218d-bf1f-4e7e-bb5d-b805af003246\") " pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l"
Oct 30 22:06:57.411535 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.411297    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/87f77489-c825-4f2c-a6ce-70e06f68f187-var-lib-kubelet\") pod \"tuned-dr9c2\" (UID: \"87f77489-c825-4f2c-a6ce-70e06f68f187\") " pod="openshift-cluster-node-tuning-operator/tuned-dr9c2"
Oct 30 22:06:57.413279 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.413168    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9ea63ef4-6b84-46e6-a753-bca60280734a-metrics-tls\") pod \"dns-default-c749j\" (UID: \"9ea63ef4-6b84-46e6-a753-bca60280734a\") " pod="openshift-dns/dns-default-c749j"
Oct 30 22:06:57.489918 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.489897    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-c749j"
Oct 30 22:06:57.813312 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.813252    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-c749j"]
Oct 30 22:06:57.819066 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:06:57.819006    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ea63ef4_6b84_46e6_a753_bca60280734a.slice/crio-f909955dbb75626bcb73ad4bcf7253e271fe52664740f69ebd12f55d84faf59e WatchSource:0}: Error finding container f909955dbb75626bcb73ad4bcf7253e271fe52664740f69ebd12f55d84faf59e: Status 404 returned error can't find the container with id f909955dbb75626bcb73ad4bcf7253e271fe52664740f69ebd12f55d84faf59e
Oct 30 22:06:57.821643 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.821627    4477 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 22:06:57.914648 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.914622    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe4c45fd-690b-4111-9d15-93dc183b3e46-metrics-certs\") pod \"network-metrics-daemon-6hhw9\" (UID: \"fe4c45fd-690b-4111-9d15-93dc183b3e46\") " pod="openshift-multus/network-metrics-daemon-6hhw9"
Oct 30 22:06:57.914722 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:57.914669    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b529dc48-9c57-4a0e-a4c0-49f53556ae34-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-69blh\" (UID: \"b529dc48-9c57-4a0e-a4c0-49f53556ae34\") " pod="openshift-insights/insights-runtime-extractor-69blh"
Oct 30 22:06:57.914791 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:06:57.914774    4477 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 22:06:57.914834 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:06:57.914822    4477 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 22:06:57.914834 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:06:57.914830    4477 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe4c45fd-690b-4111-9d15-93dc183b3e46-metrics-certs podName:fe4c45fd-690b-4111-9d15-93dc183b3e46 nodeName:}" failed. No retries permitted until 2026-10-30 22:06:58.914815838 +0000 UTC m=+2.840742058 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fe4c45fd-690b-4111-9d15-93dc183b3e46-metrics-certs") pod "network-metrics-daemon-6hhw9" (UID: "fe4c45fd-690b-4111-9d15-93dc183b3e46") : secret "metrics-daemon-secret" not found
Oct 30 22:06:57.914898 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:06:57.914861    4477 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b529dc48-9c57-4a0e-a4c0-49f53556ae34-insights-runtime-extractor-tls podName:b529dc48-9c57-4a0e-a4c0-49f53556ae34 nodeName:}" failed. No retries permitted until 2026-10-30 22:06:58.91485099 +0000 UTC m=+2.840777193 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b529dc48-9c57-4a0e-a4c0-49f53556ae34-insights-runtime-extractor-tls") pod "insights-runtime-extractor-69blh" (UID: "b529dc48-9c57-4a0e-a4c0-49f53556ae34") : secret "insights-runtime-extractor-tls" not found
Oct 30 22:06:58.225181 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:58.225149    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-c749j" event={"ID":"9ea63ef4-6b84-46e6-a753-bca60280734a","Type":"ContainerStarted","Data":"f909955dbb75626bcb73ad4bcf7253e271fe52664740f69ebd12f55d84faf59e"}
Oct 30 22:06:58.432938 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:58.432918    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l"
Oct 30 22:06:58.519856 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:58.519805    4477 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-b74lk_e9c20fdf-bea5-4d6c-b252-b29b08bfdf63/node-ca/0.log"
Oct 30 22:06:58.921797 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:58.921763    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe4c45fd-690b-4111-9d15-93dc183b3e46-metrics-certs\") pod \"network-metrics-daemon-6hhw9\" (UID: \"fe4c45fd-690b-4111-9d15-93dc183b3e46\") " pod="openshift-multus/network-metrics-daemon-6hhw9"
Oct 30 22:06:58.921935 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:58.921813    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b529dc48-9c57-4a0e-a4c0-49f53556ae34-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-69blh\" (UID: \"b529dc48-9c57-4a0e-a4c0-49f53556ae34\") " pod="openshift-insights/insights-runtime-extractor-69blh"
Oct 30 22:06:58.921935 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:06:58.921915    4477 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 22:06:58.922053 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:06:58.921915    4477 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 22:06:58.922053 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:06:58.922018    4477 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe4c45fd-690b-4111-9d15-93dc183b3e46-metrics-certs podName:fe4c45fd-690b-4111-9d15-93dc183b3e46 nodeName:}" failed. No retries permitted until 2026-10-30 22:07:00.921991534 +0000 UTC m=+4.847917742 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fe4c45fd-690b-4111-9d15-93dc183b3e46-metrics-certs") pod "network-metrics-daemon-6hhw9" (UID: "fe4c45fd-690b-4111-9d15-93dc183b3e46") : secret "metrics-daemon-secret" not found
Oct 30 22:06:58.922053 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:06:58.922051    4477 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b529dc48-9c57-4a0e-a4c0-49f53556ae34-insights-runtime-extractor-tls podName:b529dc48-9c57-4a0e-a4c0-49f53556ae34 nodeName:}" failed. No retries permitted until 2026-10-30 22:07:00.922028182 +0000 UTC m=+4.847954366 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b529dc48-9c57-4a0e-a4c0-49f53556ae34-insights-runtime-extractor-tls") pod "insights-runtime-extractor-69blh" (UID: "b529dc48-9c57-4a0e-a4c0-49f53556ae34") : secret "insights-runtime-extractor-tls" not found
Oct 30 22:06:59.519758 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:59.519714    4477 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-wp8vp_98075172-cd56-41f2-8203-fc2639e95df9/serve-healthcheck-canary/0.log"
Oct 30 22:06:59.919634 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:06:59.919607    4477 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-nzx7v_d4e7cd84-8a3f-4ef7-970c-008a9c895a47/migrator/0.log"
Oct 30 22:07:00.232260 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:00.232194    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-c749j" event={"ID":"9ea63ef4-6b84-46e6-a753-bca60280734a","Type":"ContainerStarted","Data":"bcf6fca318ffab7f42a4000951716dba3426dd28ae75360465d6eaafda3da9c0"}
Oct 30 22:07:00.232260 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:00.232252    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-c749j" event={"ID":"9ea63ef4-6b84-46e6-a753-bca60280734a","Type":"ContainerStarted","Data":"cc727cf3a0c3c477f6e228e6b78c138ce85eb867e4ba85c828c502eff5e79c62"}
Oct 30 22:07:00.232440 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:00.232345    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-c749j"
Oct 30 22:07:00.933844 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:00.933807    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b529dc48-9c57-4a0e-a4c0-49f53556ae34-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-69blh\" (UID: \"b529dc48-9c57-4a0e-a4c0-49f53556ae34\") " pod="openshift-insights/insights-runtime-extractor-69blh"
Oct 30 22:07:00.934336 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:00.933860    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe4c45fd-690b-4111-9d15-93dc183b3e46-metrics-certs\") pod \"network-metrics-daemon-6hhw9\" (UID: \"fe4c45fd-690b-4111-9d15-93dc183b3e46\") " pod="openshift-multus/network-metrics-daemon-6hhw9"
Oct 30 22:07:00.934336 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:07:00.933953    4477 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 22:07:00.934336 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:07:00.933963    4477 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 22:07:00.934336 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:07:00.934010    4477 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe4c45fd-690b-4111-9d15-93dc183b3e46-metrics-certs podName:fe4c45fd-690b-4111-9d15-93dc183b3e46 nodeName:}" failed. No retries permitted until 2026-10-30 22:07:04.933994712 +0000 UTC m=+8.859920896 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fe4c45fd-690b-4111-9d15-93dc183b3e46-metrics-certs") pod "network-metrics-daemon-6hhw9" (UID: "fe4c45fd-690b-4111-9d15-93dc183b3e46") : secret "metrics-daemon-secret" not found
Oct 30 22:07:00.934336 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:07:00.934024    4477 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b529dc48-9c57-4a0e-a4c0-49f53556ae34-insights-runtime-extractor-tls podName:b529dc48-9c57-4a0e-a4c0-49f53556ae34 nodeName:}" failed. No retries permitted until 2026-10-30 22:07:04.934017817 +0000 UTC m=+8.859944000 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b529dc48-9c57-4a0e-a4c0-49f53556ae34-insights-runtime-extractor-tls") pod "insights-runtime-extractor-69blh" (UID: "b529dc48-9c57-4a0e-a4c0-49f53556ae34") : secret "insights-runtime-extractor-tls" not found
Oct 30 22:07:01.055587 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:01.055563    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l"
Oct 30 22:07:01.055788 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:01.055776    4477 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 22:07:03.541691 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:03.541658    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jcf8l"
Oct 30 22:07:03.839520 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:03.839441    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-vs5pt"
Oct 30 22:07:04.957637 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:04.957597    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b529dc48-9c57-4a0e-a4c0-49f53556ae34-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-69blh\" (UID: \"b529dc48-9c57-4a0e-a4c0-49f53556ae34\") " pod="openshift-insights/insights-runtime-extractor-69blh"
Oct 30 22:07:04.957637 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:04.957654    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe4c45fd-690b-4111-9d15-93dc183b3e46-metrics-certs\") pod \"network-metrics-daemon-6hhw9\" (UID: \"fe4c45fd-690b-4111-9d15-93dc183b3e46\") " pod="openshift-multus/network-metrics-daemon-6hhw9"
Oct 30 22:07:04.958129 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:07:04.957738    4477 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 22:07:04.958129 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:07:04.957789    4477 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe4c45fd-690b-4111-9d15-93dc183b3e46-metrics-certs podName:fe4c45fd-690b-4111-9d15-93dc183b3e46 nodeName:}" failed. No retries permitted until 2026-10-30 22:07:12.957775541 +0000 UTC m=+16.883701724 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fe4c45fd-690b-4111-9d15-93dc183b3e46-metrics-certs") pod "network-metrics-daemon-6hhw9" (UID: "fe4c45fd-690b-4111-9d15-93dc183b3e46") : secret "metrics-daemon-secret" not found
Oct 30 22:07:04.959964 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:04.959931    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b529dc48-9c57-4a0e-a4c0-49f53556ae34-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-69blh\" (UID: \"b529dc48-9c57-4a0e-a4c0-49f53556ae34\") " pod="openshift-insights/insights-runtime-extractor-69blh"
Oct 30 22:07:04.993581 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:04.993560    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-69blh"
Oct 30 22:07:05.293179 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:05.293112    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-69blh"]
Oct 30 22:07:05.299339 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:07:05.299314    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb529dc48_9c57_4a0e_a4c0_49f53556ae34.slice/crio-3978b6e7a6bf83235ccdde7c766926e49526dae0c3bdf2ec29c170ed062e0d26 WatchSource:0}: Error finding container 3978b6e7a6bf83235ccdde7c766926e49526dae0c3bdf2ec29c170ed062e0d26: Status 404 returned error can't find the container with id 3978b6e7a6bf83235ccdde7c766926e49526dae0c3bdf2ec29c170ed062e0d26
Oct 30 22:07:06.248857 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:06.248827    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-69blh" event={"ID":"b529dc48-9c57-4a0e-a4c0-49f53556ae34","Type":"ContainerStarted","Data":"5628114d7269f07c6acee1cd785f5a0d6ed3e4c32f571828c319a5f68c0b0218"}
Oct 30 22:07:06.249261 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:06.248886    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-69blh" event={"ID":"b529dc48-9c57-4a0e-a4c0-49f53556ae34","Type":"ContainerStarted","Data":"e4566ff9d259b1753ac252bee44012076f26dd39e340834377fb3f06e39c191c"}
Oct 30 22:07:06.249261 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:06.248906    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-69blh" event={"ID":"b529dc48-9c57-4a0e-a4c0-49f53556ae34","Type":"ContainerStarted","Data":"3978b6e7a6bf83235ccdde7c766926e49526dae0c3bdf2ec29c170ed062e0d26"}
Oct 30 22:07:06.805737 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:06.805705    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-mm655"
Oct 30 22:07:07.255406 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:07.255375    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-69blh" event={"ID":"b529dc48-9c57-4a0e-a4c0-49f53556ae34","Type":"ContainerStarted","Data":"2ec2bebc1ec704a77442235f4e9898764750d692848278635b7cce3ef6a86687"}
Oct 30 22:07:09.934191 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:09.934156    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-q284j"]
Oct 30 22:07:09.936299 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:09.936279    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-q284j"
Oct 30 22:07:09.951135 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:09.951114    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Oct 30 22:07:09.961833 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:09.961811    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-xnq4q\""
Oct 30 22:07:09.962263 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:09.962245    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-q284j"]
Oct 30 22:07:10.092909 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:10.092876    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/819fb22e-2858-4cbe-9735-5f98f2b53cbc-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-q284j\" (UID: \"819fb22e-2858-4cbe-9735-5f98f2b53cbc\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-q284j"
Oct 30 22:07:10.193977 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:10.193914    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/819fb22e-2858-4cbe-9735-5f98f2b53cbc-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-q284j\" (UID: \"819fb22e-2858-4cbe-9735-5f98f2b53cbc\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-q284j"
Oct 30 22:07:10.196153 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:10.196132    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/819fb22e-2858-4cbe-9735-5f98f2b53cbc-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-q284j\" (UID: \"819fb22e-2858-4cbe-9735-5f98f2b53cbc\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-q284j"
Oct 30 22:07:10.236619 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:10.236589    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-c749j"
Oct 30 22:07:10.246146 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:10.246129    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-q284j"
Oct 30 22:07:10.532772 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:10.532703    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-q284j"]
Oct 30 22:07:10.538386 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:07:10.538363    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod819fb22e_2858_4cbe_9735_5f98f2b53cbc.slice/crio-ffacae63a52f92a3f8d72d55f1de65f7da32ffe015bcb54d874b06b31d47ac67 WatchSource:0}: Error finding container ffacae63a52f92a3f8d72d55f1de65f7da32ffe015bcb54d874b06b31d47ac67: Status 404 returned error can't find the container with id ffacae63a52f92a3f8d72d55f1de65f7da32ffe015bcb54d874b06b31d47ac67
Oct 30 22:07:11.268282 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:11.268243    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-q284j" event={"ID":"819fb22e-2858-4cbe-9735-5f98f2b53cbc","Type":"ContainerStarted","Data":"ffacae63a52f92a3f8d72d55f1de65f7da32ffe015bcb54d874b06b31d47ac67"}
Oct 30 22:07:12.272256 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:12.272221    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-q284j" event={"ID":"819fb22e-2858-4cbe-9735-5f98f2b53cbc","Type":"ContainerStarted","Data":"de33155c5bbb1029013392e59eea7a8f5bb4da9d90149475bafc06c3cd9a0972"}
Oct 30 22:07:12.272681 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:12.272430    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-q284j"
Oct 30 22:07:12.277143 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:12.277119    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-q284j"
Oct 30 22:07:12.308222 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:12.308181    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-q284j" podStartSLOduration=2.4203130760000002 podStartE2EDuration="3.30817033s" totalImagesPullingTime="887.857254ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:12.307856171 +0000 UTC m=+16.233782377" watchObservedRunningTime="2026-10-30 22:07:12.30817033 +0000 UTC m=+16.234096527"
Oct 30 22:07:12.657788 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:12.657753    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-rcb97"]
Oct 30 22:07:12.660059 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:12.660022    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-rcb97"
Oct 30 22:07:12.673563 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:12.673538    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Oct 30 22:07:12.673658 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:12.673597    4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 22:07:12.673658 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:12.673632    4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 22:07:12.684139 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:12.684115    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Oct 30 22:07:12.684208 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:12.684139    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-vsj8z\""
Oct 30 22:07:12.684242 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:12.684204    4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 22:07:12.684922 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:12.684906    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-rcb97"]
Oct 30 22:07:12.812414 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:12.812387    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4dc828b5-9f84-47db-95c9-24e3ae008789-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-rcb97\" (UID: \"4dc828b5-9f84-47db-95c9-24e3ae008789\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rcb97"
Oct 30 22:07:12.812528 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:12.812418    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqlxr\" (UniqueName: \"kubernetes.io/projected/4dc828b5-9f84-47db-95c9-24e3ae008789-kube-api-access-jqlxr\") pod \"prometheus-operator-9cd89b74c-rcb97\" (UID: \"4dc828b5-9f84-47db-95c9-24e3ae008789\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rcb97"
Oct 30 22:07:12.812528 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:12.812438    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/4dc828b5-9f84-47db-95c9-24e3ae008789-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-rcb97\" (UID: \"4dc828b5-9f84-47db-95c9-24e3ae008789\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rcb97"
Oct 30 22:07:12.812528 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:12.812458    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4dc828b5-9f84-47db-95c9-24e3ae008789-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-rcb97\" (UID: \"4dc828b5-9f84-47db-95c9-24e3ae008789\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rcb97"
Oct 30 22:07:12.913636 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:12.913576    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4dc828b5-9f84-47db-95c9-24e3ae008789-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-rcb97\" (UID: \"4dc828b5-9f84-47db-95c9-24e3ae008789\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rcb97"
Oct 30 22:07:12.913722 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:12.913632    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4dc828b5-9f84-47db-95c9-24e3ae008789-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-rcb97\" (UID: \"4dc828b5-9f84-47db-95c9-24e3ae008789\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rcb97"
Oct 30 22:07:12.913722 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:12.913664    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqlxr\" (UniqueName: \"kubernetes.io/projected/4dc828b5-9f84-47db-95c9-24e3ae008789-kube-api-access-jqlxr\") pod \"prometheus-operator-9cd89b74c-rcb97\" (UID: \"4dc828b5-9f84-47db-95c9-24e3ae008789\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rcb97"
Oct 30 22:07:12.913722 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:12.913692    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/4dc828b5-9f84-47db-95c9-24e3ae008789-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-rcb97\" (UID: \"4dc828b5-9f84-47db-95c9-24e3ae008789\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rcb97"
Oct 30 22:07:12.914357 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:12.914340    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4dc828b5-9f84-47db-95c9-24e3ae008789-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-rcb97\" (UID: \"4dc828b5-9f84-47db-95c9-24e3ae008789\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rcb97"
Oct 30 22:07:12.916052 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:12.916018    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/4dc828b5-9f84-47db-95c9-24e3ae008789-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-rcb97\" (UID: \"4dc828b5-9f84-47db-95c9-24e3ae008789\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rcb97"
Oct 30 22:07:12.916155 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:12.916025    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4dc828b5-9f84-47db-95c9-24e3ae008789-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-rcb97\" (UID: \"4dc828b5-9f84-47db-95c9-24e3ae008789\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rcb97"
Oct 30 22:07:12.931250 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:12.931225    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqlxr\" (UniqueName: \"kubernetes.io/projected/4dc828b5-9f84-47db-95c9-24e3ae008789-kube-api-access-jqlxr\") pod \"prometheus-operator-9cd89b74c-rcb97\" (UID: \"4dc828b5-9f84-47db-95c9-24e3ae008789\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-rcb97"
Oct 30 22:07:12.971146 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:12.971116    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-rcb97"
Oct 30 22:07:13.014986 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:13.014953    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe4c45fd-690b-4111-9d15-93dc183b3e46-metrics-certs\") pod \"network-metrics-daemon-6hhw9\" (UID: \"fe4c45fd-690b-4111-9d15-93dc183b3e46\") " pod="openshift-multus/network-metrics-daemon-6hhw9"
Oct 30 22:07:13.017813 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:13.017792    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe4c45fd-690b-4111-9d15-93dc183b3e46-metrics-certs\") pod \"network-metrics-daemon-6hhw9\" (UID: \"fe4c45fd-690b-4111-9d15-93dc183b3e46\") " pod="openshift-multus/network-metrics-daemon-6hhw9"
Oct 30 22:07:13.086087 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:13.086063    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6hhw9"
Oct 30 22:07:13.262977 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:07:13.262946    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dc828b5_9f84_47db_95c9_24e3ae008789.slice/crio-167f72988c7be3aa527263e7f6f267b12bd251e98f03a92e62b555d892d3f274 WatchSource:0}: Error finding container 167f72988c7be3aa527263e7f6f267b12bd251e98f03a92e62b555d892d3f274: Status 404 returned error can't find the container with id 167f72988c7be3aa527263e7f6f267b12bd251e98f03a92e62b555d892d3f274
Oct 30 22:07:13.263257 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:13.263238    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-rcb97"]
Oct 30 22:07:13.275833 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:13.275803    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-rcb97" event={"ID":"4dc828b5-9f84-47db-95c9-24e3ae008789","Type":"ContainerStarted","Data":"167f72988c7be3aa527263e7f6f267b12bd251e98f03a92e62b555d892d3f274"}
Oct 30 22:07:13.370856 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:13.370833    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6hhw9"]
Oct 30 22:07:13.375339 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:07:13.375320    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe4c45fd_690b_4111_9d15_93dc183b3e46.slice/crio-bfa7df6b6fb285635ac1cc5cad93b379375bcdc88eb3eacd02c4cf13d99516c7 WatchSource:0}: Error finding container bfa7df6b6fb285635ac1cc5cad93b379375bcdc88eb3eacd02c4cf13d99516c7: Status 404 returned error can't find the container with id bfa7df6b6fb285635ac1cc5cad93b379375bcdc88eb3eacd02c4cf13d99516c7
Oct 30 22:07:14.281190 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:14.281155    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6hhw9" event={"ID":"fe4c45fd-690b-4111-9d15-93dc183b3e46","Type":"ContainerStarted","Data":"bfa7df6b6fb285635ac1cc5cad93b379375bcdc88eb3eacd02c4cf13d99516c7"}
Oct 30 22:07:15.289806 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:15.289766    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6hhw9" event={"ID":"fe4c45fd-690b-4111-9d15-93dc183b3e46","Type":"ContainerStarted","Data":"b6be7d41284ae23186a870d3b6a66fc872ade68b9c2c0c0d95d9643697f5f552"}
Oct 30 22:07:15.290289 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:15.289845    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6hhw9" event={"ID":"fe4c45fd-690b-4111-9d15-93dc183b3e46","Type":"ContainerStarted","Data":"2c6d79262a19cba22f38df22a8dcec43580609818de616878ba1c5092d65c37a"}
Oct 30 22:07:15.291698 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:15.291674    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-rcb97" event={"ID":"4dc828b5-9f84-47db-95c9-24e3ae008789","Type":"ContainerStarted","Data":"f077a17889e7f3303253dcd1f80c49943ba6a8feec8fa522626da46aae44ad56"}
Oct 30 22:07:15.291801 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:15.291716    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-rcb97" event={"ID":"4dc828b5-9f84-47db-95c9-24e3ae008789","Type":"ContainerStarted","Data":"c1063b8a641749dc9ddb7ea103c8eb51aac238aa7e8363add1fe7f9c99a80e85"}
Oct 30 22:07:15.361158 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:15.360632    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-rcb97" podStartSLOduration=2.059476553 podStartE2EDuration="3.360609431s" totalImagesPullingTime="1.301132878s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:15.36008116 +0000 UTC m=+19.286007366" watchObservedRunningTime="2026-10-30 22:07:15.360609431 +0000 UTC m=+19.286535631"
Oct 30 22:07:17.251536 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:17.251503    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-p6d59"]
Oct 30 22:07:17.254134 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:17.254111    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-p6d59"
Oct 30 22:07:17.277398 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:17.274626    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Oct 30 22:07:17.277398 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:17.274724    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Oct 30 22:07:17.284986 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:17.284963    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-wgqc2\""
Oct 30 22:07:17.285804 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:17.285784    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-p6d59"]
Oct 30 22:07:17.301552 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:17.301533    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-bd84r"]
Oct 30 22:07:17.303586 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:17.303568    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bd84r"
Oct 30 22:07:17.321134 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:17.321108    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-r45ks\""
Oct 30 22:07:17.321134 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:17.321119    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Oct 30 22:07:17.321134 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:17.321116    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Oct 30 22:07:17.321594 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:17.321578    4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Oct 30 22:07:17.322631 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:17.322615    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-7cwrt"]
Oct 30 22:07:17.324672 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:17.324655    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7cwrt"
Oct 30 22:07:17.331197 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:17.331067    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-bd84r"]
Oct 30 22:07:17.344950 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:17.344927    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/de5a375a-a179-450f-b21d-982bfa72e489-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-p6d59\" (UID: \"de5a375a-a179-450f-b21d-982bfa72e489\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-p6d59"
Oct 30 22:07:17.345030 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:17.344957    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/de5a375a-a179-450f-b21d-982bfa72e489-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-p6d59\" (UID: \"de5a375a-a179-450f-b21d-982bfa72e489\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-p6d59"
Oct 30 22:07:17.345030 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:17.344986    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/de5a375a-a179-450f-b21d-982bfa72e489-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-p6d59\" (UID: \"de5a375a-a179-450f-b21d-982bfa72e489\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-p6d59"
Oct 30 22:07:17.345121 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:17.345099    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jv2jd\" (UniqueName: \"kubernetes.io/projected/de5a375a-a179-450f-b21d-982bfa72e489-kube-api-access-jv2jd\") pod \"openshift-state-metrics-7bd7489957-p6d59\" (UID: \"de5a375a-a179-450f-b21d-982bfa72e489\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-p6d59"
Oct 30 22:07:17.355053 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:17.355026    4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 22:07:17.374686 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:17.374659    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 22:07:17.374793 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:17.374708    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 22:07:17.374793 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:17.374730    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-btsx8\""
Oct 30 22:07:17.445520 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:17.445491    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/de5a375a-a179-450f-b21d-982bfa72e489-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-p6d59\" (UID: \"de5a375a-a179-450f-b21d-982bfa72e489\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-p6d59"
Oct 30 22:07:17.445624 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:17.445530    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/46d31fbe-a529-46d2-9385-1bf960a9fb85-metrics-client-ca\") pod \"node-exporter-7cwrt\" (UID: \"46d31fbe-a529-46d2-9385-1bf960a9fb85\") " pod="openshift-monitoring/node-exporter-7cwrt"
Oct 30 22:07:17.445624 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:17.445559    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c0550643-5300-4f60-ad13-ac4504c60a13-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-bd84r\" (UID: \"c0550643-5300-4f60-ad13-ac4504c60a13\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bd84r"
Oct 30 22:07:17.445624 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:17.445589    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/46d31fbe-a529-46d2-9385-1bf960a9fb85-node-exporter-wtmp\") pod \"node-exporter-7cwrt\" (UID: \"46d31fbe-a529-46d2-9385-1bf960a9fb85\") " pod="openshift-monitoring/node-exporter-7cwrt"
Oct 30 22:07:17.445774 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:17.445645    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5pjh\" (UniqueName: \"kubernetes.io/projected/c0550643-5300-4f60-ad13-ac4504c60a13-kube-api-access-t5pjh\") pod \"kube-state-metrics-5dc8d7d89b-bd84r\" (UID: \"c0550643-5300-4f60-ad13-ac4504c60a13\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bd84r"
Oct 30 22:07:17.445774 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:17.445721    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/46d31fbe-a529-46d2-9385-1bf960a9fb85-sys\") pod \"node-exporter-7cwrt\" (UID: \"46d31fbe-a529-46d2-9385-1bf960a9fb85\") " pod="openshift-monitoring/node-exporter-7cwrt"
Oct 30 22:07:17.445774 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:17.445741    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/46d31fbe-a529-46d2-9385-1bf960a9fb85-node-exporter-tls\") pod \"node-exporter-7cwrt\" (UID: \"46d31fbe-a529-46d2-9385-1bf960a9fb85\") " pod="openshift-monitoring/node-exporter-7cwrt"
Oct 30 22:07:17.445774 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:17.445770    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/c0550643-5300-4f60-ad13-ac4504c60a13-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-bd84r\" (UID: \"c0550643-5300-4f60-ad13-ac4504c60a13\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bd84r"
Oct 30 22:07:17.446215 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:17.445828    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/46d31fbe-a529-46d2-9385-1bf960a9fb85-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7cwrt\" (UID: \"46d31fbe-a529-46d2-9385-1bf960a9fb85\") " pod="openshift-monitoring/node-exporter-7cwrt"
Oct 30 22:07:17.446215 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:17.445851    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c0550643-5300-4f60-ad13-ac4504c60a13-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-bd84r\" (UID: \"c0550643-5300-4f60-ad13-ac4504c60a13\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bd84r"
Oct 30 22:07:17.446215 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:17.445874    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jv2jd\" (UniqueName: \"kubernetes.io/projected/de5a375a-a179-450f-b21d-982bfa72e489-kube-api-access-jv2jd\") pod \"openshift-state-metrics-7bd7489957-p6d59\" (UID: \"de5a375a-a179-450f-b21d-982bfa72e489\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-p6d59"
Oct 30 22:07:17.446215 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:17.445892    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/46d31fbe-a529-46d2-9385-1bf960a9fb85-node-exporter-textfile\") pod \"node-exporter-7cwrt\" (UID: \"46d31fbe-a529-46d2-9385-1bf960a9fb85\") " pod="openshift-monitoring/node-exporter-7cwrt"
Oct 30 22:07:17.446215 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:17.445908    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qxg7\" (UniqueName: \"kubernetes.io/projected/46d31fbe-a529-46d2-9385-1bf960a9fb85-kube-api-access-5qxg7\") pod \"node-exporter-7cwrt\" (UID: \"46d31fbe-a529-46d2-9385-1bf960a9fb85\") " pod="openshift-monitoring/node-exporter-7cwrt"
Oct 30 22:07:17.446215 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:17.445954    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/46d31fbe-a529-46d2-9385-1bf960a9fb85-node-exporter-accelerators-collector-config\") pod \"node-exporter-7cwrt\" (UID: \"46d31fbe-a529-46d2-9385-1bf960a9fb85\") " pod="openshift-monitoring/node-exporter-7cwrt"
Oct 30 22:07:17.446215 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:17.445974    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/de5a375a-a179-450f-b21d-982bfa72e489-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-p6d59\" (UID: \"de5a375a-a179-450f-b21d-982bfa72e489\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-p6d59"
Oct 30 22:07:17.446215 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:17.445994    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/46d31fbe-a529-46d2-9385-1bf960a9fb85-root\") pod \"node-exporter-7cwrt\" (UID: \"46d31fbe-a529-46d2-9385-1bf960a9fb85\") " pod="openshift-monitoring/node-exporter-7cwrt"
Oct 30 22:07:17.446215 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:17.446157    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/de5a375a-a179-450f-b21d-982bfa72e489-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-p6d59\" (UID: \"de5a375a-a179-450f-b21d-982bfa72e489\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-p6d59"
Oct 30 22:07:17.446497 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:17.446220    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c0550643-5300-4f60-ad13-ac4504c60a13-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-bd84r\" (UID: \"c0550643-5300-4f60-ad13-ac4504c60a13\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bd84r"
Oct 30 22:07:17.446497 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:17.446314    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/c0550643-5300-4f60-ad13-ac4504c60a13-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-bd84r\" (UID: \"c0550643-5300-4f60-ad13-ac4504c60a13\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bd84r"
Oct 30 22:07:17.446686 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:17.446670    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/de5a375a-a179-450f-b21d-982bfa72e489-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-p6d59\" (UID: \"de5a375a-a179-450f-b21d-982bfa72e489\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-p6d59"
Oct 30 22:07:17.447978 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:17.447962    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/de5a375a-a179-450f-b21d-982bfa72e489-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-p6d59\" (UID: \"de5a375a-a179-450f-b21d-982bfa72e489\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-p6d59"
Oct 30 22:07:17.448428 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:17.448413    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/de5a375a-a179-450f-b21d-982bfa72e489-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-p6d59\" (UID: \"de5a375a-a179-450f-b21d-982bfa72e489\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-p6d59"
Oct 30 22:07:17.464862 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:17.464841    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jv2jd\" (UniqueName: \"kubernetes.io/projected/de5a375a-a179-450f-b21d-982bfa72e489-kube-api-access-jv2jd\") pod \"openshift-state-metrics-7bd7489957-p6d59\" (UID: \"de5a375a-a179-450f-b21d-982bfa72e489\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-p6d59"
Oct 30 22:07:17.546817 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:17.546761    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/46d31fbe-a529-46d2-9385-1bf960a9fb85-node-exporter-tls\") pod \"node-exporter-7cwrt\" (UID: \"46d31fbe-a529-46d2-9385-1bf960a9fb85\") " pod="openshift-monitoring/node-exporter-7cwrt"
Oct 30 22:07:17.546817 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:17.546794    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/c0550643-5300-4f60-ad13-ac4504c60a13-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-bd84r\" (UID: \"c0550643-5300-4f60-ad13-ac4504c60a13\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bd84r"
Oct 30 22:07:17.546951 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:17.546827    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/46d31fbe-a529-46d2-9385-1bf960a9fb85-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7cwrt\" (UID: \"46d31fbe-a529-46d2-9385-1bf960a9fb85\") " pod="openshift-monitoring/node-exporter-7cwrt"
Oct 30 22:07:17.546951 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:17.546850    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c0550643-5300-4f60-ad13-ac4504c60a13-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-bd84r\" (UID: \"c0550643-5300-4f60-ad13-ac4504c60a13\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bd84r"
Oct 30 22:07:17.546951 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:17.546880    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/46d31fbe-a529-46d2-9385-1bf960a9fb85-node-exporter-textfile\") pod \"node-exporter-7cwrt\" (UID: \"46d31fbe-a529-46d2-9385-1bf960a9fb85\") " pod="openshift-monitoring/node-exporter-7cwrt"
Oct 30 22:07:17.547122 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:17.546967    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qxg7\" (UniqueName: \"kubernetes.io/projected/46d31fbe-a529-46d2-9385-1bf960a9fb85-kube-api-access-5qxg7\") pod \"node-exporter-7cwrt\" (UID: \"46d31fbe-a529-46d2-9385-1bf960a9fb85\") " pod="openshift-monitoring/node-exporter-7cwrt"
Oct 30 22:07:17.547122 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:17.547004    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/46d31fbe-a529-46d2-9385-1bf960a9fb85-node-exporter-accelerators-collector-config\") pod \"node-exporter-7cwrt\" (UID: \"46d31fbe-a529-46d2-9385-1bf960a9fb85\") " pod="openshift-monitoring/node-exporter-7cwrt"
Oct 30 22:07:17.547122 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:17.547031    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/46d31fbe-a529-46d2-9385-1bf960a9fb85-root\") pod \"node-exporter-7cwrt\" (UID: \"46d31fbe-a529-46d2-9385-1bf960a9fb85\") " pod="openshift-monitoring/node-exporter-7cwrt"
Oct 30 22:07:17.547122 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:17.547081    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c0550643-5300-4f60-ad13-ac4504c60a13-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-bd84r\" (UID: \"c0550643-5300-4f60-ad13-ac4504c60a13\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bd84r"
Oct 30 22:07:17.547122 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:17.547108    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/c0550643-5300-4f60-ad13-ac4504c60a13-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-bd84r\" (UID: \"c0550643-5300-4f60-ad13-ac4504c60a13\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bd84r"
Oct 30 22:07:17.547359 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:17.547141    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/46d31fbe-a529-46d2-9385-1bf960a9fb85-metrics-client-ca\") pod \"node-exporter-7cwrt\" (UID: \"46d31fbe-a529-46d2-9385-1bf960a9fb85\") " pod="openshift-monitoring/node-exporter-7cwrt"
Oct 30 22:07:17.547359 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:17.547169    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c0550643-5300-4f60-ad13-ac4504c60a13-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-bd84r\" (UID: \"c0550643-5300-4f60-ad13-ac4504c60a13\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bd84r"
Oct 30 22:07:17.547359 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:17.547197    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/46d31fbe-a529-46d2-9385-1bf960a9fb85-node-exporter-wtmp\") pod \"node-exporter-7cwrt\" (UID: \"46d31fbe-a529-46d2-9385-1bf960a9fb85\") " pod="openshift-monitoring/node-exporter-7cwrt"
Oct 30 22:07:17.547359 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:17.547224    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5pjh\" (UniqueName: \"kubernetes.io/projected/c0550643-5300-4f60-ad13-ac4504c60a13-kube-api-access-t5pjh\") pod \"kube-state-metrics-5dc8d7d89b-bd84r\" (UID: \"c0550643-5300-4f60-ad13-ac4504c60a13\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bd84r"
Oct 30 22:07:17.547359 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:17.547261    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/46d31fbe-a529-46d2-9385-1bf960a9fb85-sys\") pod \"node-exporter-7cwrt\" (UID: \"46d31fbe-a529-46d2-9385-1bf960a9fb85\") " pod="openshift-monitoring/node-exporter-7cwrt"
Oct 30 22:07:17.547359 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:17.547300    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/46d31fbe-a529-46d2-9385-1bf960a9fb85-node-exporter-textfile\") pod \"node-exporter-7cwrt\" (UID: \"46d31fbe-a529-46d2-9385-1bf960a9fb85\") " pod="openshift-monitoring/node-exporter-7cwrt"
Oct 30 22:07:17.547359 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:17.547335    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/46d31fbe-a529-46d2-9385-1bf960a9fb85-sys\") pod \"node-exporter-7cwrt\" (UID: \"46d31fbe-a529-46d2-9385-1bf960a9fb85\") " pod="openshift-monitoring/node-exporter-7cwrt"
Oct 30 22:07:17.547690 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:17.547625    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/c0550643-5300-4f60-ad13-ac4504c60a13-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-bd84r\" (UID: \"c0550643-5300-4f60-ad13-ac4504c60a13\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bd84r"
Oct 30 22:07:17.547690 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:17.547630    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/46d31fbe-a529-46d2-9385-1bf960a9fb85-node-exporter-accelerators-collector-config\") pod \"node-exporter-7cwrt\" (UID: \"46d31fbe-a529-46d2-9385-1bf960a9fb85\") " pod="openshift-monitoring/node-exporter-7cwrt"
Oct 30 22:07:17.547884 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:17.547864    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/46d31fbe-a529-46d2-9385-1bf960a9fb85-metrics-client-ca\") pod \"node-exporter-7cwrt\" (UID: \"46d31fbe-a529-46d2-9385-1bf960a9fb85\") " pod="openshift-monitoring/node-exporter-7cwrt"
Oct 30 22:07:17.547973 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:17.547952    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/c0550643-5300-4f60-ad13-ac4504c60a13-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-bd84r\" (UID: \"c0550643-5300-4f60-ad13-ac4504c60a13\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bd84r"
Oct 30 22:07:17.548059 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:17.547999    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/46d31fbe-a529-46d2-9385-1bf960a9fb85-root\") pod \"node-exporter-7cwrt\" (UID: \"46d31fbe-a529-46d2-9385-1bf960a9fb85\") " pod="openshift-monitoring/node-exporter-7cwrt"
Oct 30 22:07:17.548059 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:17.548014    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/46d31fbe-a529-46d2-9385-1bf960a9fb85-node-exporter-wtmp\") pod \"node-exporter-7cwrt\" (UID: \"46d31fbe-a529-46d2-9385-1bf960a9fb85\") " pod="openshift-monitoring/node-exporter-7cwrt"
Oct 30 22:07:17.548335 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:17.548315    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c0550643-5300-4f60-ad13-ac4504c60a13-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-bd84r\" (UID: \"c0550643-5300-4f60-ad13-ac4504c60a13\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bd84r"
Oct 30 22:07:17.549323 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:17.549297    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/46d31fbe-a529-46d2-9385-1bf960a9fb85-node-exporter-tls\") pod \"node-exporter-7cwrt\" (UID: \"46d31fbe-a529-46d2-9385-1bf960a9fb85\") " pod="openshift-monitoring/node-exporter-7cwrt"
Oct 30 22:07:17.549613 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:17.549589    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c0550643-5300-4f60-ad13-ac4504c60a13-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-bd84r\" (UID: \"c0550643-5300-4f60-ad13-ac4504c60a13\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bd84r"
Oct 30 22:07:17.549749 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:17.549728    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c0550643-5300-4f60-ad13-ac4504c60a13-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-bd84r\" (UID: \"c0550643-5300-4f60-ad13-ac4504c60a13\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bd84r"
Oct 30 22:07:17.549793 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:17.549751    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/46d31fbe-a529-46d2-9385-1bf960a9fb85-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7cwrt\" (UID: \"46d31fbe-a529-46d2-9385-1bf960a9fb85\") " pod="openshift-monitoring/node-exporter-7cwrt"
Oct 30 22:07:17.566123 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:17.566104    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-p6d59"
Oct 30 22:07:17.566683 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:17.566667    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qxg7\" (UniqueName: \"kubernetes.io/projected/46d31fbe-a529-46d2-9385-1bf960a9fb85-kube-api-access-5qxg7\") pod \"node-exporter-7cwrt\" (UID: \"46d31fbe-a529-46d2-9385-1bf960a9fb85\") " pod="openshift-monitoring/node-exporter-7cwrt"
Oct 30 22:07:17.566920 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:17.566903    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5pjh\" (UniqueName: \"kubernetes.io/projected/c0550643-5300-4f60-ad13-ac4504c60a13-kube-api-access-t5pjh\") pod \"kube-state-metrics-5dc8d7d89b-bd84r\" (UID: \"c0550643-5300-4f60-ad13-ac4504c60a13\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bd84r"
Oct 30 22:07:17.615216 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:17.615192    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bd84r"
Oct 30 22:07:17.637250 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:17.637225    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7cwrt"
Oct 30 22:07:17.646946 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:07:17.646912    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46d31fbe_a529_46d2_9385_1bf960a9fb85.slice/crio-fa64b1172d39189f6a92848b3b55bbafa070eb24ffb1b860ea19f7fff1af617c WatchSource:0}: Error finding container fa64b1172d39189f6a92848b3b55bbafa070eb24ffb1b860ea19f7fff1af617c: Status 404 returned error can't find the container with id fa64b1172d39189f6a92848b3b55bbafa070eb24ffb1b860ea19f7fff1af617c
Oct 30 22:07:17.863441 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:17.863374    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-p6d59"]
Oct 30 22:07:17.869347 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:07:17.869320    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde5a375a_a179_450f_b21d_982bfa72e489.slice/crio-a91c121736a328751e95b20cc68bcc5aaaaaff05cb20ec5d34696f875873ce10 WatchSource:0}: Error finding container a91c121736a328751e95b20cc68bcc5aaaaaff05cb20ec5d34696f875873ce10: Status 404 returned error can't find the container with id a91c121736a328751e95b20cc68bcc5aaaaaff05cb20ec5d34696f875873ce10
Oct 30 22:07:17.915860 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:17.915832    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-bd84r"]
Oct 30 22:07:17.920187 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:07:17.920127    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0550643_5300_4f60_ad13_ac4504c60a13.slice/crio-3cb975bce4d2ef21bbb8e782fdd71afe78ce70acb8d9edd0bbdac3b6972e126a WatchSource:0}: Error finding container 3cb975bce4d2ef21bbb8e782fdd71afe78ce70acb8d9edd0bbdac3b6972e126a: Status 404 returned error can't find the container with id 3cb975bce4d2ef21bbb8e782fdd71afe78ce70acb8d9edd0bbdac3b6972e126a
Oct 30 22:07:18.302932 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:18.302893    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7cwrt" event={"ID":"46d31fbe-a529-46d2-9385-1bf960a9fb85","Type":"ContainerStarted","Data":"fa64b1172d39189f6a92848b3b55bbafa070eb24ffb1b860ea19f7fff1af617c"}
Oct 30 22:07:18.304251 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:18.304224    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bd84r" event={"ID":"c0550643-5300-4f60-ad13-ac4504c60a13","Type":"ContainerStarted","Data":"3cb975bce4d2ef21bbb8e782fdd71afe78ce70acb8d9edd0bbdac3b6972e126a"}
Oct 30 22:07:18.306100 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:18.306073    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-p6d59" event={"ID":"de5a375a-a179-450f-b21d-982bfa72e489","Type":"ContainerStarted","Data":"bbed33173ea2f1e5479115747bf0b10e0b08b9b64d5d216c835dfee692a8e157"}
Oct 30 22:07:18.306199 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:18.306127    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-p6d59" event={"ID":"de5a375a-a179-450f-b21d-982bfa72e489","Type":"ContainerStarted","Data":"f4169eb9d461fb81117c63a85971fbbf29a2dc5fac24d37c78e531c759ad3b85"}
Oct 30 22:07:18.306199 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:18.306153    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-p6d59" event={"ID":"de5a375a-a179-450f-b21d-982bfa72e489","Type":"ContainerStarted","Data":"a91c121736a328751e95b20cc68bcc5aaaaaff05cb20ec5d34696f875873ce10"}
Oct 30 22:07:18.770009 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:18.769975    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 22:07:18.774561 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:18.774525    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 22:07:18.790665 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:18.790637    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Oct 30 22:07:18.790824 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:18.790732    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Oct 30 22:07:18.790824 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:18.790795    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Oct 30 22:07:18.790824 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:18.790801    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Oct 30 22:07:18.790824 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:18.790811    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Oct 30 22:07:18.791084 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:18.790888    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Oct 30 22:07:18.792192 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:18.792169    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-jtfxw\""
Oct 30 22:07:18.792302 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:18.792271    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Oct 30 22:07:18.799175 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:18.799150    4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Oct 30 22:07:18.799295 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:18.799254    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Oct 30 22:07:18.808618 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:18.808475    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 22:07:18.856889 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:18.856817    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ff0958b2-307f-49c2-8dc6-30d7958b2c91-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ff0958b2-307f-49c2-8dc6-30d7958b2c91\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 22:07:18.856889 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:18.856865    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ff0958b2-307f-49c2-8dc6-30d7958b2c91-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ff0958b2-307f-49c2-8dc6-30d7958b2c91\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 22:07:18.857114 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:18.856939    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9trxh\" (UniqueName: \"kubernetes.io/projected/ff0958b2-307f-49c2-8dc6-30d7958b2c91-kube-api-access-9trxh\") pod \"alertmanager-main-0\" (UID: \"ff0958b2-307f-49c2-8dc6-30d7958b2c91\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 22:07:18.857114 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:18.857018    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ff0958b2-307f-49c2-8dc6-30d7958b2c91-config-volume\") pod \"alertmanager-main-0\" (UID: \"ff0958b2-307f-49c2-8dc6-30d7958b2c91\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 22:07:18.857114 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:18.857064    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ff0958b2-307f-49c2-8dc6-30d7958b2c91-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ff0958b2-307f-49c2-8dc6-30d7958b2c91\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 22:07:18.857264 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:18.857162    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ff0958b2-307f-49c2-8dc6-30d7958b2c91-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ff0958b2-307f-49c2-8dc6-30d7958b2c91\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 22:07:18.857264 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:18.857184    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ff0958b2-307f-49c2-8dc6-30d7958b2c91-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ff0958b2-307f-49c2-8dc6-30d7958b2c91\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 22:07:18.857264 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:18.857207    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ff0958b2-307f-49c2-8dc6-30d7958b2c91-config-out\") pod \"alertmanager-main-0\" (UID: \"ff0958b2-307f-49c2-8dc6-30d7958b2c91\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 22:07:18.857415 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:18.857276    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ff0958b2-307f-49c2-8dc6-30d7958b2c91-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ff0958b2-307f-49c2-8dc6-30d7958b2c91\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 22:07:18.857415 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:18.857301    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ff0958b2-307f-49c2-8dc6-30d7958b2c91-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ff0958b2-307f-49c2-8dc6-30d7958b2c91\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 22:07:18.857415 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:18.857331    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff0958b2-307f-49c2-8dc6-30d7958b2c91-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ff0958b2-307f-49c2-8dc6-30d7958b2c91\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 22:07:18.857415 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:18.857399    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ff0958b2-307f-49c2-8dc6-30d7958b2c91-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ff0958b2-307f-49c2-8dc6-30d7958b2c91\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 22:07:18.857607 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:18.857439    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ff0958b2-307f-49c2-8dc6-30d7958b2c91-web-config\") pod \"alertmanager-main-0\" (UID: \"ff0958b2-307f-49c2-8dc6-30d7958b2c91\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 22:07:18.958906 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:18.958862    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ff0958b2-307f-49c2-8dc6-30d7958b2c91-web-config\") pod \"alertmanager-main-0\" (UID: \"ff0958b2-307f-49c2-8dc6-30d7958b2c91\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 22:07:18.959103 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:18.958921    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ff0958b2-307f-49c2-8dc6-30d7958b2c91-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ff0958b2-307f-49c2-8dc6-30d7958b2c91\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 22:07:18.959103 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:18.958955    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ff0958b2-307f-49c2-8dc6-30d7958b2c91-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ff0958b2-307f-49c2-8dc6-30d7958b2c91\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 22:07:18.959103 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:18.958974    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9trxh\" (UniqueName: \"kubernetes.io/projected/ff0958b2-307f-49c2-8dc6-30d7958b2c91-kube-api-access-9trxh\") pod \"alertmanager-main-0\" (UID: \"ff0958b2-307f-49c2-8dc6-30d7958b2c91\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 22:07:18.959103 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:18.959000    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ff0958b2-307f-49c2-8dc6-30d7958b2c91-config-volume\") pod \"alertmanager-main-0\" (UID: \"ff0958b2-307f-49c2-8dc6-30d7958b2c91\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 22:07:18.959103 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:18.959023    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ff0958b2-307f-49c2-8dc6-30d7958b2c91-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ff0958b2-307f-49c2-8dc6-30d7958b2c91\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 22:07:18.959336 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:18.959208    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ff0958b2-307f-49c2-8dc6-30d7958b2c91-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ff0958b2-307f-49c2-8dc6-30d7958b2c91\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 22:07:18.959336 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:18.959243    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ff0958b2-307f-49c2-8dc6-30d7958b2c91-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ff0958b2-307f-49c2-8dc6-30d7958b2c91\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 22:07:18.959336 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:18.959273    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ff0958b2-307f-49c2-8dc6-30d7958b2c91-config-out\") pod \"alertmanager-main-0\" (UID: \"ff0958b2-307f-49c2-8dc6-30d7958b2c91\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 22:07:18.959336 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:18.959294    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ff0958b2-307f-49c2-8dc6-30d7958b2c91-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ff0958b2-307f-49c2-8dc6-30d7958b2c91\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 22:07:18.959336 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:18.959322    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ff0958b2-307f-49c2-8dc6-30d7958b2c91-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ff0958b2-307f-49c2-8dc6-30d7958b2c91\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 22:07:18.959566 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:18.959344    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff0958b2-307f-49c2-8dc6-30d7958b2c91-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ff0958b2-307f-49c2-8dc6-30d7958b2c91\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 22:07:18.959566 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:18.959379    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ff0958b2-307f-49c2-8dc6-30d7958b2c91-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ff0958b2-307f-49c2-8dc6-30d7958b2c91\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 22:07:18.960629 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:18.960030    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ff0958b2-307f-49c2-8dc6-30d7958b2c91-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ff0958b2-307f-49c2-8dc6-30d7958b2c91\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 22:07:18.962637 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:18.962353    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ff0958b2-307f-49c2-8dc6-30d7958b2c91-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ff0958b2-307f-49c2-8dc6-30d7958b2c91\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 22:07:18.962637 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:18.962589    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff0958b2-307f-49c2-8dc6-30d7958b2c91-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ff0958b2-307f-49c2-8dc6-30d7958b2c91\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 22:07:18.964221 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:18.964193    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ff0958b2-307f-49c2-8dc6-30d7958b2c91-config-volume\") pod \"alertmanager-main-0\" (UID: \"ff0958b2-307f-49c2-8dc6-30d7958b2c91\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 22:07:18.965178 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:18.965136    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ff0958b2-307f-49c2-8dc6-30d7958b2c91-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ff0958b2-307f-49c2-8dc6-30d7958b2c91\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 22:07:18.965605 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:18.965580    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ff0958b2-307f-49c2-8dc6-30d7958b2c91-config-out\") pod \"alertmanager-main-0\" (UID: \"ff0958b2-307f-49c2-8dc6-30d7958b2c91\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 22:07:18.965816 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:18.965795    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ff0958b2-307f-49c2-8dc6-30d7958b2c91-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ff0958b2-307f-49c2-8dc6-30d7958b2c91\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 22:07:18.966015 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:18.965994    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ff0958b2-307f-49c2-8dc6-30d7958b2c91-web-config\") pod \"alertmanager-main-0\" (UID: \"ff0958b2-307f-49c2-8dc6-30d7958b2c91\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 22:07:18.966244 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:18.966198    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ff0958b2-307f-49c2-8dc6-30d7958b2c91-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ff0958b2-307f-49c2-8dc6-30d7958b2c91\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 22:07:18.966415 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:18.966394    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ff0958b2-307f-49c2-8dc6-30d7958b2c91-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ff0958b2-307f-49c2-8dc6-30d7958b2c91\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 22:07:18.967222 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:18.967203    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ff0958b2-307f-49c2-8dc6-30d7958b2c91-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ff0958b2-307f-49c2-8dc6-30d7958b2c91\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 22:07:18.973559 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:18.973538    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ff0958b2-307f-49c2-8dc6-30d7958b2c91-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ff0958b2-307f-49c2-8dc6-30d7958b2c91\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 22:07:18.983548 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:18.983523    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9trxh\" (UniqueName: \"kubernetes.io/projected/ff0958b2-307f-49c2-8dc6-30d7958b2c91-kube-api-access-9trxh\") pod \"alertmanager-main-0\" (UID: \"ff0958b2-307f-49c2-8dc6-30d7958b2c91\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 22:07:19.099951 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:19.099929    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 22:07:19.311203 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:19.311180    4477 generic.go:356] "Generic (PLEG): container finished" podID="46d31fbe-a529-46d2-9385-1bf960a9fb85" containerID="5439ef28567a1776ea62656b3bacf740ba54555ce45a9ff45870d0313aaec603" exitCode=0
Oct 30 22:07:19.311513 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:19.311256    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7cwrt" event={"ID":"46d31fbe-a529-46d2-9385-1bf960a9fb85","Type":"ContainerDied","Data":"5439ef28567a1776ea62656b3bacf740ba54555ce45a9ff45870d0313aaec603"}
Oct 30 22:07:19.312970 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:19.312951    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bd84r" event={"ID":"c0550643-5300-4f60-ad13-ac4504c60a13","Type":"ContainerStarted","Data":"22d3e7f1207121cd91cb5b3c83ec79a7e91857993b26ed8c75f5d025f6988e92"}
Oct 30 22:07:19.315344 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:19.315315    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-p6d59" event={"ID":"de5a375a-a179-450f-b21d-982bfa72e489","Type":"ContainerStarted","Data":"5073cd2abc6598cd8bbf7a280454c4562f4031887d6dbb944ca4cf582cebe6a6"}
Oct 30 22:07:19.388859 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:19.388810    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-p6d59" podStartSLOduration=1.317991609 podStartE2EDuration="2.388796214s" totalImagesPullingTime="1.070804605s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:19.38745385 +0000 UTC m=+23.313380067" watchObservedRunningTime="2026-10-30 22:07:19.388796214 +0000 UTC m=+23.314722438"
Oct 30 22:07:19.463377 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:07:19.463344    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff0958b2_307f_49c2_8dc6_30d7958b2c91.slice/crio-79b910f090c1fb045433649114fd975781ce30b51001efe4c04ecf20f4c09a27 WatchSource:0}: Error finding container 79b910f090c1fb045433649114fd975781ce30b51001efe4c04ecf20f4c09a27: Status 404 returned error can't find the container with id 79b910f090c1fb045433649114fd975781ce30b51001efe4c04ecf20f4c09a27
Oct 30 22:07:19.465067 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:19.464914    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 22:07:20.320560 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:20.320517    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7cwrt" event={"ID":"46d31fbe-a529-46d2-9385-1bf960a9fb85","Type":"ContainerStarted","Data":"5ec683fe11bceb8fcdf9533c2ac7e0e27a5bc45a9f044be480bfae173a5b2da6"}
Oct 30 22:07:20.321004 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:20.320628    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7cwrt" event={"ID":"46d31fbe-a529-46d2-9385-1bf960a9fb85","Type":"ContainerStarted","Data":"16e7d4494582c8901c2be774d6cc18dae79b90a58584ac057f59b9121f4e087e"}
Oct 30 22:07:20.322688 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:20.322657    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bd84r" event={"ID":"c0550643-5300-4f60-ad13-ac4504c60a13","Type":"ContainerStarted","Data":"8324498ab78f1fde2537b6354d3fe365bd8db0deea570d470c52cc95fc284476"}
Oct 30 22:07:20.322824 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:20.322710    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bd84r" event={"ID":"c0550643-5300-4f60-ad13-ac4504c60a13","Type":"ContainerStarted","Data":"97cf0334e894cf9714a9b61625e01f5e886fb59df8aaa13e5f1cfeae45514b9a"}
Oct 30 22:07:20.323850 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:20.323827    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ff0958b2-307f-49c2-8dc6-30d7958b2c91","Type":"ContainerStarted","Data":"79b910f090c1fb045433649114fd975781ce30b51001efe4c04ecf20f4c09a27"}
Oct 30 22:07:20.358842 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:20.358802    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-7cwrt" podStartSLOduration=2.464236289 podStartE2EDuration="3.358790816s" totalImagesPullingTime="815.801288ms" totalInitContainerRuntime="78.753239ms" isStatefulPod=true podCreationTimestamp="2026-10-30 22:07:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:20.358389595 +0000 UTC m=+24.284315804" watchObservedRunningTime="2026-10-30 22:07:20.358790816 +0000 UTC m=+24.284717016"
Oct 30 22:07:20.396730 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:20.396689    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bd84r" podStartSLOduration=2.058744245 podStartE2EDuration="3.396676433s" totalImagesPullingTime="1.337932188s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:20.396036999 +0000 UTC m=+24.321963208" watchObservedRunningTime="2026-10-30 22:07:20.396676433 +0000 UTC m=+24.322602639"
Oct 30 22:07:21.328649 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:21.328608    4477 generic.go:356] "Generic (PLEG): container finished" podID="ff0958b2-307f-49c2-8dc6-30d7958b2c91" containerID="ed9c90533fb6507613ac9ee5aea4549c0abf8aabb46ebc6367c11781ac251772" exitCode=0
Oct 30 22:07:21.329082 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:21.328704    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ff0958b2-307f-49c2-8dc6-30d7958b2c91","Type":"ContainerDied","Data":"ed9c90533fb6507613ac9ee5aea4549c0abf8aabb46ebc6367c11781ac251772"}
Oct 30 22:07:21.857253 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:21.857225    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-c78d588bd-vf9gh"]
Oct 30 22:07:21.859814 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:21.859791    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-c78d588bd-vf9gh"
Oct 30 22:07:21.873547 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:21.873525    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-wr88t\""
Oct 30 22:07:21.873823 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:21.873802    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Oct 30 22:07:21.883991 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:21.883969    4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Oct 30 22:07:21.883991 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:21.883983    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-9ltderoslumhv\""
Oct 30 22:07:21.884155 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:21.884069    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Oct 30 22:07:21.884155 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:21.884106    4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Oct 30 22:07:21.888270 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:21.888250    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-c78d588bd-vf9gh"]
Oct 30 22:07:21.983800 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:21.983769    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/c7a30fb6-9f07-4ade-82d3-d3f00f02476f-secret-metrics-server-tls\") pod \"metrics-server-c78d588bd-vf9gh\" (UID: \"c7a30fb6-9f07-4ade-82d3-d3f00f02476f\") " pod="openshift-monitoring/metrics-server-c78d588bd-vf9gh"
Oct 30 22:07:21.983939 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:21.983814    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c7a30fb6-9f07-4ade-82d3-d3f00f02476f-client-ca-bundle\") pod \"metrics-server-c78d588bd-vf9gh\" (UID: \"c7a30fb6-9f07-4ade-82d3-d3f00f02476f\") " pod="openshift-monitoring/metrics-server-c78d588bd-vf9gh"
Oct 30 22:07:21.983939 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:21.983847    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/c7a30fb6-9f07-4ade-82d3-d3f00f02476f-secret-metrics-server-client-certs\") pod \"metrics-server-c78d588bd-vf9gh\" (UID: \"c7a30fb6-9f07-4ade-82d3-d3f00f02476f\") " pod="openshift-monitoring/metrics-server-c78d588bd-vf9gh"
Oct 30 22:07:21.984023 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:21.984003    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/c7a30fb6-9f07-4ade-82d3-d3f00f02476f-audit-log\") pod \"metrics-server-c78d588bd-vf9gh\" (UID: \"c7a30fb6-9f07-4ade-82d3-d3f00f02476f\") " pod="openshift-monitoring/metrics-server-c78d588bd-vf9gh"
Oct 30 22:07:21.984085 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:21.984058    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tk9lr\" (UniqueName: \"kubernetes.io/projected/c7a30fb6-9f07-4ade-82d3-d3f00f02476f-kube-api-access-tk9lr\") pod \"metrics-server-c78d588bd-vf9gh\" (UID: \"c7a30fb6-9f07-4ade-82d3-d3f00f02476f\") " pod="openshift-monitoring/metrics-server-c78d588bd-vf9gh"
Oct 30 22:07:21.984189 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:21.984171    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/c7a30fb6-9f07-4ade-82d3-d3f00f02476f-metrics-server-audit-profiles\") pod \"metrics-server-c78d588bd-vf9gh\" (UID: \"c7a30fb6-9f07-4ade-82d3-d3f00f02476f\") " pod="openshift-monitoring/metrics-server-c78d588bd-vf9gh"
Oct 30 22:07:21.984223 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:21.984204    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c7a30fb6-9f07-4ade-82d3-d3f00f02476f-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-c78d588bd-vf9gh\" (UID: \"c7a30fb6-9f07-4ade-82d3-d3f00f02476f\") " pod="openshift-monitoring/metrics-server-c78d588bd-vf9gh"
Oct 30 22:07:22.085355 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:22.085326    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/c7a30fb6-9f07-4ade-82d3-d3f00f02476f-metrics-server-audit-profiles\") pod \"metrics-server-c78d588bd-vf9gh\" (UID: \"c7a30fb6-9f07-4ade-82d3-d3f00f02476f\") " pod="openshift-monitoring/metrics-server-c78d588bd-vf9gh"
Oct 30 22:07:22.085508 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:22.085360    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c7a30fb6-9f07-4ade-82d3-d3f00f02476f-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-c78d588bd-vf9gh\" (UID: \"c7a30fb6-9f07-4ade-82d3-d3f00f02476f\") " pod="openshift-monitoring/metrics-server-c78d588bd-vf9gh"
Oct 30 22:07:22.085508 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:22.085387    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/c7a30fb6-9f07-4ade-82d3-d3f00f02476f-secret-metrics-server-tls\") pod \"metrics-server-c78d588bd-vf9gh\" (UID: \"c7a30fb6-9f07-4ade-82d3-d3f00f02476f\") " pod="openshift-monitoring/metrics-server-c78d588bd-vf9gh"
Oct 30 22:07:22.085508 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:22.085420    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c7a30fb6-9f07-4ade-82d3-d3f00f02476f-client-ca-bundle\") pod \"metrics-server-c78d588bd-vf9gh\" (UID: \"c7a30fb6-9f07-4ade-82d3-d3f00f02476f\") " pod="openshift-monitoring/metrics-server-c78d588bd-vf9gh"
Oct 30 22:07:22.085508 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:22.085452    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/c7a30fb6-9f07-4ade-82d3-d3f00f02476f-secret-metrics-server-client-certs\") pod \"metrics-server-c78d588bd-vf9gh\" (UID: \"c7a30fb6-9f07-4ade-82d3-d3f00f02476f\") " pod="openshift-monitoring/metrics-server-c78d588bd-vf9gh"
Oct 30 22:07:22.085508 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:22.085481    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/c7a30fb6-9f07-4ade-82d3-d3f00f02476f-audit-log\") pod \"metrics-server-c78d588bd-vf9gh\" (UID: \"c7a30fb6-9f07-4ade-82d3-d3f00f02476f\") " pod="openshift-monitoring/metrics-server-c78d588bd-vf9gh"
Oct 30 22:07:22.085776 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:22.085742    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tk9lr\" (UniqueName: \"kubernetes.io/projected/c7a30fb6-9f07-4ade-82d3-d3f00f02476f-kube-api-access-tk9lr\") pod \"metrics-server-c78d588bd-vf9gh\" (UID: \"c7a30fb6-9f07-4ade-82d3-d3f00f02476f\") " pod="openshift-monitoring/metrics-server-c78d588bd-vf9gh"
Oct 30 22:07:22.085945 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:22.085914    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/c7a30fb6-9f07-4ade-82d3-d3f00f02476f-audit-log\") pod \"metrics-server-c78d588bd-vf9gh\" (UID: \"c7a30fb6-9f07-4ade-82d3-d3f00f02476f\") " pod="openshift-monitoring/metrics-server-c78d588bd-vf9gh"
Oct 30 22:07:22.086239 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:22.086215    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c7a30fb6-9f07-4ade-82d3-d3f00f02476f-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-c78d588bd-vf9gh\" (UID: \"c7a30fb6-9f07-4ade-82d3-d3f00f02476f\") " pod="openshift-monitoring/metrics-server-c78d588bd-vf9gh"
Oct 30 22:07:22.086462 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:22.086440    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/c7a30fb6-9f07-4ade-82d3-d3f00f02476f-metrics-server-audit-profiles\") pod \"metrics-server-c78d588bd-vf9gh\" (UID: \"c7a30fb6-9f07-4ade-82d3-d3f00f02476f\") " pod="openshift-monitoring/metrics-server-c78d588bd-vf9gh"
Oct 30 22:07:22.088535 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:22.088512    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/c7a30fb6-9f07-4ade-82d3-d3f00f02476f-secret-metrics-server-tls\") pod \"metrics-server-c78d588bd-vf9gh\" (UID: \"c7a30fb6-9f07-4ade-82d3-d3f00f02476f\") " pod="openshift-monitoring/metrics-server-c78d588bd-vf9gh"
Oct 30 22:07:22.088673 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:22.088652    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/c7a30fb6-9f07-4ade-82d3-d3f00f02476f-secret-metrics-server-client-certs\") pod \"metrics-server-c78d588bd-vf9gh\" (UID: \"c7a30fb6-9f07-4ade-82d3-d3f00f02476f\") " pod="openshift-monitoring/metrics-server-c78d588bd-vf9gh"
Oct 30 22:07:22.088738 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:22.088653    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/c7a30fb6-9f07-4ade-82d3-d3f00f02476f-client-ca-bundle\") pod \"metrics-server-c78d588bd-vf9gh\" (UID: \"c7a30fb6-9f07-4ade-82d3-d3f00f02476f\") " pod="openshift-monitoring/metrics-server-c78d588bd-vf9gh"
Oct 30 22:07:22.103964 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:22.103923    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tk9lr\" (UniqueName: \"kubernetes.io/projected/c7a30fb6-9f07-4ade-82d3-d3f00f02476f-kube-api-access-tk9lr\") pod \"metrics-server-c78d588bd-vf9gh\" (UID: \"c7a30fb6-9f07-4ade-82d3-d3f00f02476f\") " pod="openshift-monitoring/metrics-server-c78d588bd-vf9gh"
Oct 30 22:07:22.105967 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:22.105948    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-shq24"]
Oct 30 22:07:22.109198 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:22.109157    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-shq24"
Oct 30 22:07:22.123349 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:22.123326    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Oct 30 22:07:22.123589 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:22.123571    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-fnrpx\""
Oct 30 22:07:22.130033 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:22.129984    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-shq24"]
Oct 30 22:07:22.171651 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:22.171626    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-c78d588bd-vf9gh"
Oct 30 22:07:22.186852 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:22.186827    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4dc9807e-ebe7-4789-b0db-6a09ce597ed2-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-shq24\" (UID: \"4dc9807e-ebe7-4789-b0db-6a09ce597ed2\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-shq24"
Oct 30 22:07:22.186984 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:22.186935    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpqhv\" (UniqueName: \"kubernetes.io/projected/4dc9807e-ebe7-4789-b0db-6a09ce597ed2-kube-api-access-tpqhv\") pod \"monitoring-plugin-77f4f76594-shq24\" (UID: \"4dc9807e-ebe7-4789-b0db-6a09ce597ed2\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-shq24"
Oct 30 22:07:22.287841 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:22.287810    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4dc9807e-ebe7-4789-b0db-6a09ce597ed2-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-shq24\" (UID: \"4dc9807e-ebe7-4789-b0db-6a09ce597ed2\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-shq24"
Oct 30 22:07:22.288015 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:22.287873    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpqhv\" (UniqueName: \"kubernetes.io/projected/4dc9807e-ebe7-4789-b0db-6a09ce597ed2-kube-api-access-tpqhv\") pod \"monitoring-plugin-77f4f76594-shq24\" (UID: \"4dc9807e-ebe7-4789-b0db-6a09ce597ed2\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-shq24"
Oct 30 22:07:22.290550 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:22.290522    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4dc9807e-ebe7-4789-b0db-6a09ce597ed2-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-shq24\" (UID: \"4dc9807e-ebe7-4789-b0db-6a09ce597ed2\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-shq24"
Oct 30 22:07:22.306216 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:22.306189    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpqhv\" (UniqueName: \"kubernetes.io/projected/4dc9807e-ebe7-4789-b0db-6a09ce597ed2-kube-api-access-tpqhv\") pod \"monitoring-plugin-77f4f76594-shq24\" (UID: \"4dc9807e-ebe7-4789-b0db-6a09ce597ed2\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-shq24"
Oct 30 22:07:22.421244 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:22.421169    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-shq24"
Oct 30 22:07:22.491646 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:22.491536    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-c78d588bd-vf9gh"]
Oct 30 22:07:22.566131 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:22.566099    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-69dd59c556-ssvjv"]
Oct 30 22:07:22.569547 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:22.569526    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-69dd59c556-ssvjv"
Oct 30 22:07:22.583726 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:22.583707    4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Oct 30 22:07:22.595871 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:22.595849    4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\""
Oct 30 22:07:22.606421 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:22.606400    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-69dd59c556-ssvjv"]
Oct 30 22:07:22.615530 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:22.615507    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-cj477\""
Oct 30 22:07:22.615684 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:22.615655    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Oct 30 22:07:22.615776 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:22.615761    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Oct 30 22:07:22.615911 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:22.615869    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Oct 30 22:07:22.616027 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:22.615952    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Oct 30 22:07:22.691892 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:22.691708    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcpvq\" (UniqueName: \"kubernetes.io/projected/beb3893f-2dc3-4c93-b03c-8d7c35e47a89-kube-api-access-bcpvq\") pod \"telemeter-client-69dd59c556-ssvjv\" (UID: \"beb3893f-2dc3-4c93-b03c-8d7c35e47a89\") " pod="openshift-monitoring/telemeter-client-69dd59c556-ssvjv"
Oct 30 22:07:22.691892 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:22.691781    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/beb3893f-2dc3-4c93-b03c-8d7c35e47a89-telemeter-trusted-ca-bundle\") pod \"telemeter-client-69dd59c556-ssvjv\" (UID: \"beb3893f-2dc3-4c93-b03c-8d7c35e47a89\") " pod="openshift-monitoring/telemeter-client-69dd59c556-ssvjv"
Oct 30 22:07:22.692134 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:22.691907    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/beb3893f-2dc3-4c93-b03c-8d7c35e47a89-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-69dd59c556-ssvjv\" (UID: \"beb3893f-2dc3-4c93-b03c-8d7c35e47a89\") " pod="openshift-monitoring/telemeter-client-69dd59c556-ssvjv"
Oct 30 22:07:22.692134 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:22.691963    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/beb3893f-2dc3-4c93-b03c-8d7c35e47a89-serving-certs-ca-bundle\") pod \"telemeter-client-69dd59c556-ssvjv\" (UID: \"beb3893f-2dc3-4c93-b03c-8d7c35e47a89\") " pod="openshift-monitoring/telemeter-client-69dd59c556-ssvjv"
Oct 30 22:07:22.692134 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:22.692003    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/beb3893f-2dc3-4c93-b03c-8d7c35e47a89-metrics-client-ca\") pod \"telemeter-client-69dd59c556-ssvjv\" (UID: \"beb3893f-2dc3-4c93-b03c-8d7c35e47a89\") " pod="openshift-monitoring/telemeter-client-69dd59c556-ssvjv"
Oct 30 22:07:22.692134 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:22.692073    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/beb3893f-2dc3-4c93-b03c-8d7c35e47a89-telemeter-client-tls\") pod \"telemeter-client-69dd59c556-ssvjv\" (UID: \"beb3893f-2dc3-4c93-b03c-8d7c35e47a89\") " pod="openshift-monitoring/telemeter-client-69dd59c556-ssvjv"
Oct 30 22:07:22.692134 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:22.692104    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/beb3893f-2dc3-4c93-b03c-8d7c35e47a89-secret-telemeter-client\") pod \"telemeter-client-69dd59c556-ssvjv\" (UID: \"beb3893f-2dc3-4c93-b03c-8d7c35e47a89\") " pod="openshift-monitoring/telemeter-client-69dd59c556-ssvjv"
Oct 30 22:07:22.692134 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:22.692132    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/beb3893f-2dc3-4c93-b03c-8d7c35e47a89-federate-client-tls\") pod \"telemeter-client-69dd59c556-ssvjv\" (UID: \"beb3893f-2dc3-4c93-b03c-8d7c35e47a89\") " pod="openshift-monitoring/telemeter-client-69dd59c556-ssvjv"
Oct 30 22:07:22.729747 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:22.729718    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-shq24"]
Oct 30 22:07:22.743614 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:07:22.743579    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dc9807e_ebe7_4789_b0db_6a09ce597ed2.slice/crio-2df679e163eed94f86ed6b72fbbba6d7f11c0852fdb7841c73752cffb201104d WatchSource:0}: Error finding container 2df679e163eed94f86ed6b72fbbba6d7f11c0852fdb7841c73752cffb201104d: Status 404 returned error can't find the container with id 2df679e163eed94f86ed6b72fbbba6d7f11c0852fdb7841c73752cffb201104d
Oct 30 22:07:22.792444 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:22.792423    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/beb3893f-2dc3-4c93-b03c-8d7c35e47a89-telemeter-client-tls\") pod \"telemeter-client-69dd59c556-ssvjv\" (UID: \"beb3893f-2dc3-4c93-b03c-8d7c35e47a89\") " pod="openshift-monitoring/telemeter-client-69dd59c556-ssvjv"
Oct 30 22:07:22.792563 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:22.792461    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/beb3893f-2dc3-4c93-b03c-8d7c35e47a89-secret-telemeter-client\") pod \"telemeter-client-69dd59c556-ssvjv\" (UID: \"beb3893f-2dc3-4c93-b03c-8d7c35e47a89\") " pod="openshift-monitoring/telemeter-client-69dd59c556-ssvjv"
Oct 30 22:07:22.792563 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:22.792493    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/beb3893f-2dc3-4c93-b03c-8d7c35e47a89-federate-client-tls\") pod \"telemeter-client-69dd59c556-ssvjv\" (UID: \"beb3893f-2dc3-4c93-b03c-8d7c35e47a89\") " pod="openshift-monitoring/telemeter-client-69dd59c556-ssvjv"
Oct 30 22:07:22.792563 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:22.792529    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcpvq\" (UniqueName: \"kubernetes.io/projected/beb3893f-2dc3-4c93-b03c-8d7c35e47a89-kube-api-access-bcpvq\") pod \"telemeter-client-69dd59c556-ssvjv\" (UID: \"beb3893f-2dc3-4c93-b03c-8d7c35e47a89\") " pod="openshift-monitoring/telemeter-client-69dd59c556-ssvjv"
Oct 30 22:07:22.792730 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:22.792580    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/beb3893f-2dc3-4c93-b03c-8d7c35e47a89-telemeter-trusted-ca-bundle\") pod \"telemeter-client-69dd59c556-ssvjv\" (UID: \"beb3893f-2dc3-4c93-b03c-8d7c35e47a89\") " pod="openshift-monitoring/telemeter-client-69dd59c556-ssvjv"
Oct 30 22:07:22.792730 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:22.792629    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/beb3893f-2dc3-4c93-b03c-8d7c35e47a89-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-69dd59c556-ssvjv\" (UID: \"beb3893f-2dc3-4c93-b03c-8d7c35e47a89\") " pod="openshift-monitoring/telemeter-client-69dd59c556-ssvjv"
Oct 30 22:07:22.792730 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:22.792672    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/beb3893f-2dc3-4c93-b03c-8d7c35e47a89-serving-certs-ca-bundle\") pod \"telemeter-client-69dd59c556-ssvjv\" (UID: \"beb3893f-2dc3-4c93-b03c-8d7c35e47a89\") " pod="openshift-monitoring/telemeter-client-69dd59c556-ssvjv"
Oct 30 22:07:22.792730 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:22.792706    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/beb3893f-2dc3-4c93-b03c-8d7c35e47a89-metrics-client-ca\") pod \"telemeter-client-69dd59c556-ssvjv\" (UID: \"beb3893f-2dc3-4c93-b03c-8d7c35e47a89\") " pod="openshift-monitoring/telemeter-client-69dd59c556-ssvjv"
Oct 30 22:07:22.793655 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:22.793572    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/beb3893f-2dc3-4c93-b03c-8d7c35e47a89-metrics-client-ca\") pod \"telemeter-client-69dd59c556-ssvjv\" (UID: \"beb3893f-2dc3-4c93-b03c-8d7c35e47a89\") " pod="openshift-monitoring/telemeter-client-69dd59c556-ssvjv"
Oct 30 22:07:22.793764 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:22.793704    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/beb3893f-2dc3-4c93-b03c-8d7c35e47a89-serving-certs-ca-bundle\") pod \"telemeter-client-69dd59c556-ssvjv\" (UID: \"beb3893f-2dc3-4c93-b03c-8d7c35e47a89\") " pod="openshift-monitoring/telemeter-client-69dd59c556-ssvjv"
Oct 30 22:07:22.793884 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:22.793857    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/beb3893f-2dc3-4c93-b03c-8d7c35e47a89-telemeter-trusted-ca-bundle\") pod \"telemeter-client-69dd59c556-ssvjv\" (UID: \"beb3893f-2dc3-4c93-b03c-8d7c35e47a89\") " pod="openshift-monitoring/telemeter-client-69dd59c556-ssvjv"
Oct 30 22:07:22.794982 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:22.794953    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/beb3893f-2dc3-4c93-b03c-8d7c35e47a89-telemeter-client-tls\") pod \"telemeter-client-69dd59c556-ssvjv\" (UID: \"beb3893f-2dc3-4c93-b03c-8d7c35e47a89\") " pod="openshift-monitoring/telemeter-client-69dd59c556-ssvjv"
Oct 30 22:07:22.795100 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:22.795017    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/beb3893f-2dc3-4c93-b03c-8d7c35e47a89-federate-client-tls\") pod \"telemeter-client-69dd59c556-ssvjv\" (UID: \"beb3893f-2dc3-4c93-b03c-8d7c35e47a89\") " pod="openshift-monitoring/telemeter-client-69dd59c556-ssvjv"
Oct 30 22:07:22.795292 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:22.795276    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/beb3893f-2dc3-4c93-b03c-8d7c35e47a89-secret-telemeter-client\") pod \"telemeter-client-69dd59c556-ssvjv\" (UID: \"beb3893f-2dc3-4c93-b03c-8d7c35e47a89\") " pod="openshift-monitoring/telemeter-client-69dd59c556-ssvjv"
Oct 30 22:07:22.795453 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:22.795428    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/beb3893f-2dc3-4c93-b03c-8d7c35e47a89-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-69dd59c556-ssvjv\" (UID: \"beb3893f-2dc3-4c93-b03c-8d7c35e47a89\") " pod="openshift-monitoring/telemeter-client-69dd59c556-ssvjv"
Oct 30 22:07:22.813699 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:22.813671    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcpvq\" (UniqueName: \"kubernetes.io/projected/beb3893f-2dc3-4c93-b03c-8d7c35e47a89-kube-api-access-bcpvq\") pod \"telemeter-client-69dd59c556-ssvjv\" (UID: \"beb3893f-2dc3-4c93-b03c-8d7c35e47a89\") " pod="openshift-monitoring/telemeter-client-69dd59c556-ssvjv"
Oct 30 22:07:22.883356 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:22.883332    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-69dd59c556-ssvjv"
Oct 30 22:07:23.206224 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:23.205931    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-69dd59c556-ssvjv"]
Oct 30 22:07:23.210932 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:07:23.210903    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbeb3893f_2dc3_4c93_b03c_8d7c35e47a89.slice/crio-2e25acdc1ab0e5950b726d2c8b343ab64f1840bb553cc5e44eb6960248065ce1 WatchSource:0}: Error finding container 2e25acdc1ab0e5950b726d2c8b343ab64f1840bb553cc5e44eb6960248065ce1: Status 404 returned error can't find the container with id 2e25acdc1ab0e5950b726d2c8b343ab64f1840bb553cc5e44eb6960248065ce1
Oct 30 22:07:23.344342 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:23.344305    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ff0958b2-307f-49c2-8dc6-30d7958b2c91","Type":"ContainerStarted","Data":"c7c37e921f5dc77a58159775fc3f32410bfbe37ebd09609e8d3cc2eeb3b31a99"}
Oct 30 22:07:23.344464 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:23.344392    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ff0958b2-307f-49c2-8dc6-30d7958b2c91","Type":"ContainerStarted","Data":"30da140289e5e2937a2dc54e4141d3650d66b3b64ab0e49a8e3de2fdb7213fac"}
Oct 30 22:07:23.344464 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:23.344436    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ff0958b2-307f-49c2-8dc6-30d7958b2c91","Type":"ContainerStarted","Data":"fc2add1d153b0c4608d745a1d587f530ccf5a3c6bda7dd5d5508002d135d5ac4"}
Oct 30 22:07:23.344539 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:23.344474    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ff0958b2-307f-49c2-8dc6-30d7958b2c91","Type":"ContainerStarted","Data":"94238bcd412582d3c1364e0a1d4c2dff953a9d36e46cc3a847f4d7d03c536222"}
Oct 30 22:07:23.344539 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:23.344512    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ff0958b2-307f-49c2-8dc6-30d7958b2c91","Type":"ContainerStarted","Data":"b9989a6be10d37ab3c63475e7b987bf6bf195b0da0f26b807b467b486e8aa2c7"}
Oct 30 22:07:23.346398 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:23.346363    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-c78d588bd-vf9gh" event={"ID":"c7a30fb6-9f07-4ade-82d3-d3f00f02476f","Type":"ContainerStarted","Data":"5c8b90aed1a29a7e6a8be265a0a387021055fed5c1e98fb9fa257ccc45e39205"}
Oct 30 22:07:23.347840 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:23.347815    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-69dd59c556-ssvjv" event={"ID":"beb3893f-2dc3-4c93-b03c-8d7c35e47a89","Type":"ContainerStarted","Data":"2e25acdc1ab0e5950b726d2c8b343ab64f1840bb553cc5e44eb6960248065ce1"}
Oct 30 22:07:23.349361 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:23.349337    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-shq24" event={"ID":"4dc9807e-ebe7-4789-b0db-6a09ce597ed2","Type":"ContainerStarted","Data":"2df679e163eed94f86ed6b72fbbba6d7f11c0852fdb7841c73752cffb201104d"}
Oct 30 22:07:24.359377 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:24.359340    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 22:07:24.390740 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:24.390705    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ff0958b2-307f-49c2-8dc6-30d7958b2c91","Type":"ContainerStarted","Data":"78603695f0a26f9b692426086e27124b57f7a1b424a4290084e8f6a51bee6a64"}
Oct 30 22:07:24.390906 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:24.390867    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:07:24.404934 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:24.404908    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 22:07:24.412936 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:24.412870    4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Oct 30 22:07:24.412936 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:24.412917    4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Oct 30 22:07:24.412936 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:24.412925    4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Oct 30 22:07:24.432802 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:24.432775    4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Oct 30 22:07:24.433673 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:24.432990    4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Oct 30 22:07:24.433673 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:24.433133    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Oct 30 22:07:24.437958 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:24.437523    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Oct 30 22:07:24.437958 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:24.437558    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Oct 30 22:07:24.437958 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:24.437575    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Oct 30 22:07:24.437958 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:24.437675    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Oct 30 22:07:24.437958 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:24.437699    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Oct 30 22:07:24.437958 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:24.437738    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-26f0kjcpid9an\""
Oct 30 22:07:24.437958 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:24.437807    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Oct 30 22:07:24.437958 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:24.437846    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Oct 30 22:07:24.437958 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:24.437936    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-lghk9\""
Oct 30 22:07:24.439022 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:24.439004    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Oct 30 22:07:24.440644 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:24.439407    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.204031577 podStartE2EDuration="6.439392057s" totalImagesPullingTime="3.223830852s" totalInitContainerRuntime="11.529628ms" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:18 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:24.438079547 +0000 UTC m=+28.364005772" watchObservedRunningTime="2026-10-30 22:07:24.439392057 +0000 UTC m=+28.365318266"
Oct 30 22:07:24.519636 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:24.518924    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f904edd0-10dc-4958-8d9b-9e012c1319db-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f904edd0-10dc-4958-8d9b-9e012c1319db\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:07:24.519636 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:24.518987    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f904edd0-10dc-4958-8d9b-9e012c1319db-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f904edd0-10dc-4958-8d9b-9e012c1319db\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:07:24.521105 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:24.519032    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f904edd0-10dc-4958-8d9b-9e012c1319db-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f904edd0-10dc-4958-8d9b-9e012c1319db\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:07:24.521105 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:24.520203    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9h2cn\" (UniqueName: \"kubernetes.io/projected/f904edd0-10dc-4958-8d9b-9e012c1319db-kube-api-access-9h2cn\") pod \"prometheus-k8s-0\" (UID: \"f904edd0-10dc-4958-8d9b-9e012c1319db\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:07:24.521105 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:24.520241    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f904edd0-10dc-4958-8d9b-9e012c1319db-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f904edd0-10dc-4958-8d9b-9e012c1319db\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:07:24.521105 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:24.520292    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f904edd0-10dc-4958-8d9b-9e012c1319db-config\") pod \"prometheus-k8s-0\" (UID: \"f904edd0-10dc-4958-8d9b-9e012c1319db\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:07:24.521105 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:24.520318    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f904edd0-10dc-4958-8d9b-9e012c1319db-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f904edd0-10dc-4958-8d9b-9e012c1319db\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:07:24.521105 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:24.520343    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f904edd0-10dc-4958-8d9b-9e012c1319db-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f904edd0-10dc-4958-8d9b-9e012c1319db\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:07:24.521105 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:24.520388    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f904edd0-10dc-4958-8d9b-9e012c1319db-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f904edd0-10dc-4958-8d9b-9e012c1319db\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:07:24.521105 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:24.520465    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f904edd0-10dc-4958-8d9b-9e012c1319db-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f904edd0-10dc-4958-8d9b-9e012c1319db\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:07:24.521105 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:24.520490    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f904edd0-10dc-4958-8d9b-9e012c1319db-web-config\") pod \"prometheus-k8s-0\" (UID: \"f904edd0-10dc-4958-8d9b-9e012c1319db\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:07:24.521105 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:24.520514    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f904edd0-10dc-4958-8d9b-9e012c1319db-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f904edd0-10dc-4958-8d9b-9e012c1319db\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:07:24.521105 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:24.520576    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f904edd0-10dc-4958-8d9b-9e012c1319db-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f904edd0-10dc-4958-8d9b-9e012c1319db\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:07:24.521105 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:24.520619    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f904edd0-10dc-4958-8d9b-9e012c1319db-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f904edd0-10dc-4958-8d9b-9e012c1319db\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:07:24.521105 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:24.520653    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f904edd0-10dc-4958-8d9b-9e012c1319db-config-out\") pod \"prometheus-k8s-0\" (UID: \"f904edd0-10dc-4958-8d9b-9e012c1319db\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:07:24.521105 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:24.520677    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f904edd0-10dc-4958-8d9b-9e012c1319db-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f904edd0-10dc-4958-8d9b-9e012c1319db\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:07:24.521105 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:24.520741    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f904edd0-10dc-4958-8d9b-9e012c1319db-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f904edd0-10dc-4958-8d9b-9e012c1319db\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:07:24.521105 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:24.520773    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f904edd0-10dc-4958-8d9b-9e012c1319db-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f904edd0-10dc-4958-8d9b-9e012c1319db\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:07:24.521105 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:24.520801    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f904edd0-10dc-4958-8d9b-9e012c1319db-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f904edd0-10dc-4958-8d9b-9e012c1319db\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:07:24.521105 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:24.520844    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f904edd0-10dc-4958-8d9b-9e012c1319db-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f904edd0-10dc-4958-8d9b-9e012c1319db\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:07:24.621432 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:24.621371    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f904edd0-10dc-4958-8d9b-9e012c1319db-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f904edd0-10dc-4958-8d9b-9e012c1319db\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:07:24.621432 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:24.621402    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f904edd0-10dc-4958-8d9b-9e012c1319db-config-out\") pod \"prometheus-k8s-0\" (UID: \"f904edd0-10dc-4958-8d9b-9e012c1319db\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:07:24.621432 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:24.621425    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f904edd0-10dc-4958-8d9b-9e012c1319db-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f904edd0-10dc-4958-8d9b-9e012c1319db\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:07:24.621665 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:24.621457    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f904edd0-10dc-4958-8d9b-9e012c1319db-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f904edd0-10dc-4958-8d9b-9e012c1319db\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:07:24.621665 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:24.621482    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f904edd0-10dc-4958-8d9b-9e012c1319db-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f904edd0-10dc-4958-8d9b-9e012c1319db\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:07:24.621665 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:24.621507    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f904edd0-10dc-4958-8d9b-9e012c1319db-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f904edd0-10dc-4958-8d9b-9e012c1319db\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:07:24.621665 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:24.621533    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f904edd0-10dc-4958-8d9b-9e012c1319db-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f904edd0-10dc-4958-8d9b-9e012c1319db\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:07:24.621665 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:24.621560    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f904edd0-10dc-4958-8d9b-9e012c1319db-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f904edd0-10dc-4958-8d9b-9e012c1319db\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:07:24.621665 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:24.621584    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f904edd0-10dc-4958-8d9b-9e012c1319db-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f904edd0-10dc-4958-8d9b-9e012c1319db\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:07:24.621665 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:24.621611    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f904edd0-10dc-4958-8d9b-9e012c1319db-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f904edd0-10dc-4958-8d9b-9e012c1319db\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:07:24.621665 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:24.621647    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9h2cn\" (UniqueName: \"kubernetes.io/projected/f904edd0-10dc-4958-8d9b-9e012c1319db-kube-api-access-9h2cn\") pod \"prometheus-k8s-0\" (UID: \"f904edd0-10dc-4958-8d9b-9e012c1319db\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:07:24.622091 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:24.621685    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f904edd0-10dc-4958-8d9b-9e012c1319db-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f904edd0-10dc-4958-8d9b-9e012c1319db\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:07:24.622091 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:24.621717    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f904edd0-10dc-4958-8d9b-9e012c1319db-config\") pod \"prometheus-k8s-0\" (UID: \"f904edd0-10dc-4958-8d9b-9e012c1319db\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:07:24.622091 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:24.621740    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f904edd0-10dc-4958-8d9b-9e012c1319db-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f904edd0-10dc-4958-8d9b-9e012c1319db\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:07:24.622091 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:24.621769    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f904edd0-10dc-4958-8d9b-9e012c1319db-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f904edd0-10dc-4958-8d9b-9e012c1319db\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:07:24.622091 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:24.621798    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f904edd0-10dc-4958-8d9b-9e012c1319db-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f904edd0-10dc-4958-8d9b-9e012c1319db\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:07:24.622091 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:24.621841    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f904edd0-10dc-4958-8d9b-9e012c1319db-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f904edd0-10dc-4958-8d9b-9e012c1319db\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:07:24.622091 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:24.621864    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f904edd0-10dc-4958-8d9b-9e012c1319db-web-config\") pod \"prometheus-k8s-0\" (UID: \"f904edd0-10dc-4958-8d9b-9e012c1319db\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:07:24.622091 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:24.621885    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f904edd0-10dc-4958-8d9b-9e012c1319db-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f904edd0-10dc-4958-8d9b-9e012c1319db\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:07:24.622091 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:24.621923    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f904edd0-10dc-4958-8d9b-9e012c1319db-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f904edd0-10dc-4958-8d9b-9e012c1319db\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:07:24.622091 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:24.621936    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f904edd0-10dc-4958-8d9b-9e012c1319db-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f904edd0-10dc-4958-8d9b-9e012c1319db\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:07:24.622604 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:24.622200    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f904edd0-10dc-4958-8d9b-9e012c1319db-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f904edd0-10dc-4958-8d9b-9e012c1319db\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:07:24.622604 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:24.622205    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f904edd0-10dc-4958-8d9b-9e012c1319db-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f904edd0-10dc-4958-8d9b-9e012c1319db\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:07:24.622604 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:24.622567    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f904edd0-10dc-4958-8d9b-9e012c1319db-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f904edd0-10dc-4958-8d9b-9e012c1319db\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:07:24.623095 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:24.622964    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f904edd0-10dc-4958-8d9b-9e012c1319db-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f904edd0-10dc-4958-8d9b-9e012c1319db\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:07:24.623519 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:24.623496    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f904edd0-10dc-4958-8d9b-9e012c1319db-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f904edd0-10dc-4958-8d9b-9e012c1319db\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:07:24.624574 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:24.624547    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f904edd0-10dc-4958-8d9b-9e012c1319db-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f904edd0-10dc-4958-8d9b-9e012c1319db\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:07:24.624791 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:24.624768    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f904edd0-10dc-4958-8d9b-9e012c1319db-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f904edd0-10dc-4958-8d9b-9e012c1319db\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:07:24.625023 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:24.624997    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f904edd0-10dc-4958-8d9b-9e012c1319db-config-out\") pod \"prometheus-k8s-0\" (UID: \"f904edd0-10dc-4958-8d9b-9e012c1319db\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:07:24.625967 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:24.625361    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f904edd0-10dc-4958-8d9b-9e012c1319db-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f904edd0-10dc-4958-8d9b-9e012c1319db\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:07:24.625967 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:24.625374    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f904edd0-10dc-4958-8d9b-9e012c1319db-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f904edd0-10dc-4958-8d9b-9e012c1319db\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:07:24.625967 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:24.625852    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/f904edd0-10dc-4958-8d9b-9e012c1319db-config\") pod \"prometheus-k8s-0\" (UID: \"f904edd0-10dc-4958-8d9b-9e012c1319db\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:07:24.625967 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:24.625885    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f904edd0-10dc-4958-8d9b-9e012c1319db-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f904edd0-10dc-4958-8d9b-9e012c1319db\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:07:24.626316 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:24.626293    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f904edd0-10dc-4958-8d9b-9e012c1319db-web-config\") pod \"prometheus-k8s-0\" (UID: \"f904edd0-10dc-4958-8d9b-9e012c1319db\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:07:24.627385 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:24.627346    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f904edd0-10dc-4958-8d9b-9e012c1319db-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f904edd0-10dc-4958-8d9b-9e012c1319db\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:07:24.627544 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:24.627525    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f904edd0-10dc-4958-8d9b-9e012c1319db-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f904edd0-10dc-4958-8d9b-9e012c1319db\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:07:24.627830 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:24.627804    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f904edd0-10dc-4958-8d9b-9e012c1319db-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f904edd0-10dc-4958-8d9b-9e012c1319db\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:07:24.627905 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:24.627827    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f904edd0-10dc-4958-8d9b-9e012c1319db-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f904edd0-10dc-4958-8d9b-9e012c1319db\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:07:24.628918 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:24.628897    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f904edd0-10dc-4958-8d9b-9e012c1319db-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f904edd0-10dc-4958-8d9b-9e012c1319db\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:07:24.640209 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:24.640190    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9h2cn\" (UniqueName: \"kubernetes.io/projected/f904edd0-10dc-4958-8d9b-9e012c1319db-kube-api-access-9h2cn\") pod \"prometheus-k8s-0\" (UID: \"f904edd0-10dc-4958-8d9b-9e012c1319db\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:07:24.814773 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:24.814675    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:07:25.179159 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:25.179080    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 22:07:25.183669 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:07:25.183645    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf904edd0_10dc_4958_8d9b_9e012c1319db.slice/crio-1a5bfee254bde827ac9f688534def5ec0361ccac1bf79eb00e45d3e5c073e5e9 WatchSource:0}: Error finding container 1a5bfee254bde827ac9f688534def5ec0361ccac1bf79eb00e45d3e5c073e5e9: Status 404 returned error can't find the container with id 1a5bfee254bde827ac9f688534def5ec0361ccac1bf79eb00e45d3e5c073e5e9
Oct 30 22:07:25.382010 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:25.381972    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-c78d588bd-vf9gh" event={"ID":"c7a30fb6-9f07-4ade-82d3-d3f00f02476f","Type":"ContainerStarted","Data":"6c0c20fc210faccf5c875ec43550d9c6125e678bf6f12dc1bb2b8bd8cfb2294d"}
Oct 30 22:07:25.384419 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:25.384390    4477 generic.go:356] "Generic (PLEG): container finished" podID="f904edd0-10dc-4958-8d9b-9e012c1319db" containerID="ab20b6db94f1e4cda42e0b032c7c382dea247c3c780f4399c70a7eb339cd66a6" exitCode=0
Oct 30 22:07:25.384553 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:25.384475    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f904edd0-10dc-4958-8d9b-9e012c1319db","Type":"ContainerDied","Data":"ab20b6db94f1e4cda42e0b032c7c382dea247c3c780f4399c70a7eb339cd66a6"}
Oct 30 22:07:25.384553 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:25.384528    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f904edd0-10dc-4958-8d9b-9e012c1319db","Type":"ContainerStarted","Data":"1a5bfee254bde827ac9f688534def5ec0361ccac1bf79eb00e45d3e5c073e5e9"}
Oct 30 22:07:25.388827 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:25.388798    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-shq24" event={"ID":"4dc9807e-ebe7-4789-b0db-6a09ce597ed2","Type":"ContainerStarted","Data":"b9cbbe9843224825fe4f7e1a19708d78cf53ba4464b22d994bd5ceea2dc4731f"}
Oct 30 22:07:25.391414 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:25.391385    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-shq24"
Oct 30 22:07:25.397204 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:25.397182    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-shq24"
Oct 30 22:07:25.423799 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:25.423762    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-c78d588bd-vf9gh" podStartSLOduration=2.088636825 podStartE2EDuration="4.423751037s" totalImagesPullingTime="2.335114212s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:25.423308215 +0000 UTC m=+29.349234441" watchObservedRunningTime="2026-10-30 22:07:25.423751037 +0000 UTC m=+29.349677243"
Oct 30 22:07:25.459117 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:25.459027    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-shq24" podStartSLOduration=1.362038163 podStartE2EDuration="3.459016224s" totalImagesPullingTime="2.096978061s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:25.458705326 +0000 UTC m=+29.384631551" watchObservedRunningTime="2026-10-30 22:07:25.459016224 +0000 UTC m=+29.384942431"
Oct 30 22:07:26.271409 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:07:26.271372    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 22:07:26.395089 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:26.395031    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-69dd59c556-ssvjv" event={"ID":"beb3893f-2dc3-4c93-b03c-8d7c35e47a89","Type":"ContainerStarted","Data":"947159322a424977941bb40e6fe9acfb281b1bd94d7e0ef04121f5ddd338217f"}
Oct 30 22:07:26.395478 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:26.395119    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-69dd59c556-ssvjv" event={"ID":"beb3893f-2dc3-4c93-b03c-8d7c35e47a89","Type":"ContainerStarted","Data":"1669198aa8db2cd4bc7a3f8c599c875b3d0a4c3e707201acaf0b01c4efe2da9f"}
Oct 30 22:07:26.395478 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:26.395150    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-69dd59c556-ssvjv" event={"ID":"beb3893f-2dc3-4c93-b03c-8d7c35e47a89","Type":"ContainerStarted","Data":"42c02336ceef97c47ab675f6e5cc0e8f0fbf443dbd11a980720dc0ab367c477d"}
Oct 30 22:07:26.442755 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:26.442490    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-69dd59c556-ssvjv" podStartSLOduration=2.054235922 podStartE2EDuration="4.442467451s" totalImagesPullingTime="2.388231529s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:26.440931689 +0000 UTC m=+30.366857897" watchObservedRunningTime="2026-10-30 22:07:26.442467451 +0000 UTC m=+30.368393658"
Oct 30 22:07:29.412328 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:29.412293    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f904edd0-10dc-4958-8d9b-9e012c1319db","Type":"ContainerStarted","Data":"c0c298ca2a98a892f23adecaddc83575f89fb6ed37cd952300aa86954ca586fb"}
Oct 30 22:07:29.412897 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:29.412375    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f904edd0-10dc-4958-8d9b-9e012c1319db","Type":"ContainerStarted","Data":"e3cd27a71f050df532a08814a4280c80556187adb7c8d22fa82cd343369a30e0"}
Oct 30 22:07:31.426421 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:31.426391    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f904edd0-10dc-4958-8d9b-9e012c1319db","Type":"ContainerStarted","Data":"0fe9c11ae206823bc898cf13163d98146cb190eb2952e4e250b30ed2b8b501ba"}
Oct 30 22:07:31.426700 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:31.426455    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f904edd0-10dc-4958-8d9b-9e012c1319db","Type":"ContainerStarted","Data":"6cd67108b4e7516c87b9a3cadbfd54edd7d436841f49ce876fc03d7d718ca2ec"}
Oct 30 22:07:31.426700 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:31.426482    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f904edd0-10dc-4958-8d9b-9e012c1319db","Type":"ContainerStarted","Data":"1b3367b9f00d774fd20bfb6349716fbc15f118a8d18bbfc4d0fd9af8c72c63f7"}
Oct 30 22:07:32.433710 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:32.433673    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f904edd0-10dc-4958-8d9b-9e012c1319db","Type":"ContainerStarted","Data":"e59f8a472d1f4bf287761c5eccf03ceed9f5b1b9243e84a9dd2cf9faa9c9801a"}
Oct 30 22:07:32.514699 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:32.514646    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.858605388 podStartE2EDuration="8.514633044s" totalImagesPullingTime="5.625020507s" totalInitContainerRuntime="31.007149ms" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:24 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:32.502751165 +0000 UTC m=+36.428677371" watchObservedRunningTime="2026-10-30 22:07:32.514633044 +0000 UTC m=+36.440559283"
Oct 30 22:07:33.841891 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:33.841858    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-vs5pt"
Oct 30 22:07:34.815089 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:34.815060    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:07:42.172747 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:42.172673    4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-c78d588bd-vf9gh"
Oct 30 22:07:42.173243 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:07:42.172759    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-c78d588bd-vf9gh"
Oct 30 22:07:56.270267 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:07:56.270233    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 22:08:02.177501 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:02.177467    4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-c78d588bd-vf9gh"
Oct 30 22:08:02.181266 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:02.181249    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-c78d588bd-vf9gh"
Oct 30 22:08:26.267492 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:08:26.267453    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 22:08:30.989145 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:30.989102    4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:08:31.010065 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:31.010021    4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:08:31.662773 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:31.662738    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:08:39.194005 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:39.193975    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 22:08:39.194511 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:39.194454    4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ff0958b2-307f-49c2-8dc6-30d7958b2c91" containerName="kube-rbac-proxy-web" containerID="cri-o://fc2add1d153b0c4608d745a1d587f530ccf5a3c6bda7dd5d5508002d135d5ac4" gracePeriod=120
Oct 30 22:08:39.194707 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:39.194446    4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ff0958b2-307f-49c2-8dc6-30d7958b2c91" containerName="alertmanager" containerID="cri-o://b9989a6be10d37ab3c63475e7b987bf6bf195b0da0f26b807b467b486e8aa2c7" gracePeriod=120
Oct 30 22:08:39.194987 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:39.194467    4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ff0958b2-307f-49c2-8dc6-30d7958b2c91" containerName="kube-rbac-proxy-metric" containerID="cri-o://c7c37e921f5dc77a58159775fc3f32410bfbe37ebd09609e8d3cc2eeb3b31a99" gracePeriod=120
Oct 30 22:08:39.194987 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:39.194523    4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ff0958b2-307f-49c2-8dc6-30d7958b2c91" containerName="prom-label-proxy" containerID="cri-o://78603695f0a26f9b692426086e27124b57f7a1b424a4290084e8f6a51bee6a64" gracePeriod=120
Oct 30 22:08:39.194987 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:39.194513    4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ff0958b2-307f-49c2-8dc6-30d7958b2c91" containerName="config-reloader" containerID="cri-o://94238bcd412582d3c1364e0a1d4c2dff953a9d36e46cc3a847f4d7d03c536222" gracePeriod=120
Oct 30 22:08:39.194987 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:39.194537    4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ff0958b2-307f-49c2-8dc6-30d7958b2c91" containerName="kube-rbac-proxy" containerID="cri-o://30da140289e5e2937a2dc54e4141d3650d66b3b64ab0e49a8e3de2fdb7213fac" gracePeriod=120
Oct 30 22:08:39.680205 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:39.680176    4477 generic.go:356] "Generic (PLEG): container finished" podID="ff0958b2-307f-49c2-8dc6-30d7958b2c91" containerID="78603695f0a26f9b692426086e27124b57f7a1b424a4290084e8f6a51bee6a64" exitCode=0
Oct 30 22:08:39.680205 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:39.680199    4477 generic.go:356] "Generic (PLEG): container finished" podID="ff0958b2-307f-49c2-8dc6-30d7958b2c91" containerID="30da140289e5e2937a2dc54e4141d3650d66b3b64ab0e49a8e3de2fdb7213fac" exitCode=0
Oct 30 22:08:39.680205 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:39.680206    4477 generic.go:356] "Generic (PLEG): container finished" podID="ff0958b2-307f-49c2-8dc6-30d7958b2c91" containerID="94238bcd412582d3c1364e0a1d4c2dff953a9d36e46cc3a847f4d7d03c536222" exitCode=0
Oct 30 22:08:39.680482 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:39.680212    4477 generic.go:356] "Generic (PLEG): container finished" podID="ff0958b2-307f-49c2-8dc6-30d7958b2c91" containerID="b9989a6be10d37ab3c63475e7b987bf6bf195b0da0f26b807b467b486e8aa2c7" exitCode=0
Oct 30 22:08:39.680482 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:39.680248    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ff0958b2-307f-49c2-8dc6-30d7958b2c91","Type":"ContainerDied","Data":"78603695f0a26f9b692426086e27124b57f7a1b424a4290084e8f6a51bee6a64"}
Oct 30 22:08:39.680482 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:39.680330    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ff0958b2-307f-49c2-8dc6-30d7958b2c91","Type":"ContainerDied","Data":"30da140289e5e2937a2dc54e4141d3650d66b3b64ab0e49a8e3de2fdb7213fac"}
Oct 30 22:08:39.680482 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:39.680369    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ff0958b2-307f-49c2-8dc6-30d7958b2c91","Type":"ContainerDied","Data":"94238bcd412582d3c1364e0a1d4c2dff953a9d36e46cc3a847f4d7d03c536222"}
Oct 30 22:08:39.680482 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:39.680410    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ff0958b2-307f-49c2-8dc6-30d7958b2c91","Type":"ContainerDied","Data":"b9989a6be10d37ab3c63475e7b987bf6bf195b0da0f26b807b467b486e8aa2c7"}
Oct 30 22:08:40.453960 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.453935    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 22:08:40.608301 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.608222    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff0958b2-307f-49c2-8dc6-30d7958b2c91-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ff0958b2-307f-49c2-8dc6-30d7958b2c91-metrics-client-ca\") pod \"ff0958b2-307f-49c2-8dc6-30d7958b2c91\" (UID: \"ff0958b2-307f-49c2-8dc6-30d7958b2c91\") "
Oct 30 22:08:40.608301 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.608265    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff0958b2-307f-49c2-8dc6-30d7958b2c91-tls-assets\" (UniqueName: \"kubernetes.io/projected/ff0958b2-307f-49c2-8dc6-30d7958b2c91-tls-assets\") pod \"ff0958b2-307f-49c2-8dc6-30d7958b2c91\" (UID: \"ff0958b2-307f-49c2-8dc6-30d7958b2c91\") "
Oct 30 22:08:40.608506 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.608305    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff0958b2-307f-49c2-8dc6-30d7958b2c91-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ff0958b2-307f-49c2-8dc6-30d7958b2c91-secret-alertmanager-kube-rbac-proxy-metric\") pod \"ff0958b2-307f-49c2-8dc6-30d7958b2c91\" (UID: \"ff0958b2-307f-49c2-8dc6-30d7958b2c91\") "
Oct 30 22:08:40.608506 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.608338    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff0958b2-307f-49c2-8dc6-30d7958b2c91-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ff0958b2-307f-49c2-8dc6-30d7958b2c91-secret-alertmanager-main-tls\") pod \"ff0958b2-307f-49c2-8dc6-30d7958b2c91\" (UID: \"ff0958b2-307f-49c2-8dc6-30d7958b2c91\") "
Oct 30 22:08:40.608506 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.608376    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff0958b2-307f-49c2-8dc6-30d7958b2c91-config-volume\" (UniqueName: \"kubernetes.io/secret/ff0958b2-307f-49c2-8dc6-30d7958b2c91-config-volume\") pod \"ff0958b2-307f-49c2-8dc6-30d7958b2c91\" (UID: \"ff0958b2-307f-49c2-8dc6-30d7958b2c91\") "
Oct 30 22:08:40.608506 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.608400    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff0958b2-307f-49c2-8dc6-30d7958b2c91-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ff0958b2-307f-49c2-8dc6-30d7958b2c91-alertmanager-main-db\") pod \"ff0958b2-307f-49c2-8dc6-30d7958b2c91\" (UID: \"ff0958b2-307f-49c2-8dc6-30d7958b2c91\") "
Oct 30 22:08:40.608506 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.608425    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff0958b2-307f-49c2-8dc6-30d7958b2c91-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ff0958b2-307f-49c2-8dc6-30d7958b2c91-secret-alertmanager-kube-rbac-proxy\") pod \"ff0958b2-307f-49c2-8dc6-30d7958b2c91\" (UID: \"ff0958b2-307f-49c2-8dc6-30d7958b2c91\") "
Oct 30 22:08:40.608506 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.608457    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff0958b2-307f-49c2-8dc6-30d7958b2c91-config-out\" (UniqueName: \"kubernetes.io/empty-dir/ff0958b2-307f-49c2-8dc6-30d7958b2c91-config-out\") pod \"ff0958b2-307f-49c2-8dc6-30d7958b2c91\" (UID: \"ff0958b2-307f-49c2-8dc6-30d7958b2c91\") "
Oct 30 22:08:40.608804 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.608558    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff0958b2-307f-49c2-8dc6-30d7958b2c91-metrics-client-ca" pod "ff0958b2-307f-49c2-8dc6-30d7958b2c91" (UID: "ff0958b2-307f-49c2-8dc6-30d7958b2c91"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:08:40.608804 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.608566    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff0958b2-307f-49c2-8dc6-30d7958b2c91-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ff0958b2-307f-49c2-8dc6-30d7958b2c91-secret-alertmanager-kube-rbac-proxy-web\") pod \"ff0958b2-307f-49c2-8dc6-30d7958b2c91\" (UID: \"ff0958b2-307f-49c2-8dc6-30d7958b2c91\") "
Oct 30 22:08:40.608804 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.608607    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff0958b2-307f-49c2-8dc6-30d7958b2c91-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff0958b2-307f-49c2-8dc6-30d7958b2c91-alertmanager-trusted-ca-bundle\") pod \"ff0958b2-307f-49c2-8dc6-30d7958b2c91\" (UID: \"ff0958b2-307f-49c2-8dc6-30d7958b2c91\") "
Oct 30 22:08:40.608804 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.608643    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff0958b2-307f-49c2-8dc6-30d7958b2c91-web-config\" (UniqueName: \"kubernetes.io/secret/ff0958b2-307f-49c2-8dc6-30d7958b2c91-web-config\") pod \"ff0958b2-307f-49c2-8dc6-30d7958b2c91\" (UID: \"ff0958b2-307f-49c2-8dc6-30d7958b2c91\") "
Oct 30 22:08:40.608804 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.608708    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff0958b2-307f-49c2-8dc6-30d7958b2c91-kube-api-access-9trxh\" (UniqueName: \"kubernetes.io/projected/ff0958b2-307f-49c2-8dc6-30d7958b2c91-kube-api-access-9trxh\") pod \"ff0958b2-307f-49c2-8dc6-30d7958b2c91\" (UID: \"ff0958b2-307f-49c2-8dc6-30d7958b2c91\") "
Oct 30 22:08:40.608804 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.608737    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff0958b2-307f-49c2-8dc6-30d7958b2c91-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ff0958b2-307f-49c2-8dc6-30d7958b2c91-cluster-tls-config\") pod \"ff0958b2-307f-49c2-8dc6-30d7958b2c91\" (UID: \"ff0958b2-307f-49c2-8dc6-30d7958b2c91\") "
Oct 30 22:08:40.609107 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.608975    4477 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ff0958b2-307f-49c2-8dc6-30d7958b2c91-metrics-client-ca\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:08:40.609353 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.609323    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff0958b2-307f-49c2-8dc6-30d7958b2c91-alertmanager-trusted-ca-bundle" pod "ff0958b2-307f-49c2-8dc6-30d7958b2c91" (UID: "ff0958b2-307f-49c2-8dc6-30d7958b2c91"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:08:40.609456 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.609409    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff0958b2-307f-49c2-8dc6-30d7958b2c91-alertmanager-main-db" pod "ff0958b2-307f-49c2-8dc6-30d7958b2c91" (UID: "ff0958b2-307f-49c2-8dc6-30d7958b2c91"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:08:40.612286 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.612255    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff0958b2-307f-49c2-8dc6-30d7958b2c91-config-volume" pod "ff0958b2-307f-49c2-8dc6-30d7958b2c91" (UID: "ff0958b2-307f-49c2-8dc6-30d7958b2c91"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:08:40.613784 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.613754    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff0958b2-307f-49c2-8dc6-30d7958b2c91-tls-assets" pod "ff0958b2-307f-49c2-8dc6-30d7958b2c91" (UID: "ff0958b2-307f-49c2-8dc6-30d7958b2c91"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:08:40.613989 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.613962    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff0958b2-307f-49c2-8dc6-30d7958b2c91-secret-alertmanager-kube-rbac-proxy-metric" pod "ff0958b2-307f-49c2-8dc6-30d7958b2c91" (UID: "ff0958b2-307f-49c2-8dc6-30d7958b2c91"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:08:40.613989 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.613970    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff0958b2-307f-49c2-8dc6-30d7958b2c91-kube-api-access-9trxh" pod "ff0958b2-307f-49c2-8dc6-30d7958b2c91" (UID: "ff0958b2-307f-49c2-8dc6-30d7958b2c91"). InnerVolumeSpecName "kube-api-access-9trxh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:08:40.614183 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.613985    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff0958b2-307f-49c2-8dc6-30d7958b2c91-secret-alertmanager-kube-rbac-proxy-web" pod "ff0958b2-307f-49c2-8dc6-30d7958b2c91" (UID: "ff0958b2-307f-49c2-8dc6-30d7958b2c91"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:08:40.614248 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.614208    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff0958b2-307f-49c2-8dc6-30d7958b2c91-secret-alertmanager-main-tls" pod "ff0958b2-307f-49c2-8dc6-30d7958b2c91" (UID: "ff0958b2-307f-49c2-8dc6-30d7958b2c91"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:08:40.614454 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.614436    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff0958b2-307f-49c2-8dc6-30d7958b2c91-config-out" pod "ff0958b2-307f-49c2-8dc6-30d7958b2c91" (UID: "ff0958b2-307f-49c2-8dc6-30d7958b2c91"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:08:40.614509 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.614453    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff0958b2-307f-49c2-8dc6-30d7958b2c91-secret-alertmanager-kube-rbac-proxy" pod "ff0958b2-307f-49c2-8dc6-30d7958b2c91" (UID: "ff0958b2-307f-49c2-8dc6-30d7958b2c91"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:08:40.616952 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.616925    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff0958b2-307f-49c2-8dc6-30d7958b2c91-cluster-tls-config" pod "ff0958b2-307f-49c2-8dc6-30d7958b2c91" (UID: "ff0958b2-307f-49c2-8dc6-30d7958b2c91"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:08:40.624837 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.624814    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff0958b2-307f-49c2-8dc6-30d7958b2c91-web-config" pod "ff0958b2-307f-49c2-8dc6-30d7958b2c91" (UID: "ff0958b2-307f-49c2-8dc6-30d7958b2c91"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:08:40.688194 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.688165    4477 generic.go:356] "Generic (PLEG): container finished" podID="ff0958b2-307f-49c2-8dc6-30d7958b2c91" containerID="c7c37e921f5dc77a58159775fc3f32410bfbe37ebd09609e8d3cc2eeb3b31a99" exitCode=0
Oct 30 22:08:40.688194 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.688190    4477 generic.go:356] "Generic (PLEG): container finished" podID="ff0958b2-307f-49c2-8dc6-30d7958b2c91" containerID="fc2add1d153b0c4608d745a1d587f530ccf5a3c6bda7dd5d5508002d135d5ac4" exitCode=0
Oct 30 22:08:40.688381 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.688242    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ff0958b2-307f-49c2-8dc6-30d7958b2c91","Type":"ContainerDied","Data":"c7c37e921f5dc77a58159775fc3f32410bfbe37ebd09609e8d3cc2eeb3b31a99"}
Oct 30 22:08:40.688381 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.688292    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 22:08:40.688381 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.688315    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ff0958b2-307f-49c2-8dc6-30d7958b2c91","Type":"ContainerDied","Data":"fc2add1d153b0c4608d745a1d587f530ccf5a3c6bda7dd5d5508002d135d5ac4"}
Oct 30 22:08:40.688381 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.688344    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ff0958b2-307f-49c2-8dc6-30d7958b2c91","Type":"ContainerDied","Data":"79b910f090c1fb045433649114fd975781ce30b51001efe4c04ecf20f4c09a27"}
Oct 30 22:08:40.688381 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.688379    4477 scope.go:122] "RemoveContainer" containerID="78603695f0a26f9b692426086e27124b57f7a1b424a4290084e8f6a51bee6a64"
Oct 30 22:08:40.696897 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.696877    4477 scope.go:122] "RemoveContainer" containerID="c7c37e921f5dc77a58159775fc3f32410bfbe37ebd09609e8d3cc2eeb3b31a99"
Oct 30 22:08:40.706140 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.706122    4477 scope.go:122] "RemoveContainer" containerID="30da140289e5e2937a2dc54e4141d3650d66b3b64ab0e49a8e3de2fdb7213fac"
Oct 30 22:08:40.710405 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.710385    4477 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ff0958b2-307f-49c2-8dc6-30d7958b2c91-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:08:40.710510 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.710411    4477 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ff0958b2-307f-49c2-8dc6-30d7958b2c91-secret-alertmanager-main-tls\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:08:40.710510 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.710427    4477 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ff0958b2-307f-49c2-8dc6-30d7958b2c91-config-volume\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:08:40.710510 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.710440    4477 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ff0958b2-307f-49c2-8dc6-30d7958b2c91-alertmanager-main-db\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:08:40.710510 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.710453    4477 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ff0958b2-307f-49c2-8dc6-30d7958b2c91-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:08:40.710510 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.710467    4477 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ff0958b2-307f-49c2-8dc6-30d7958b2c91-config-out\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:08:40.710510 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.710482    4477 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ff0958b2-307f-49c2-8dc6-30d7958b2c91-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:08:40.710510 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.710495    4477 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff0958b2-307f-49c2-8dc6-30d7958b2c91-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:08:40.710510 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.710508    4477 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ff0958b2-307f-49c2-8dc6-30d7958b2c91-web-config\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:08:40.710763 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.710521    4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9trxh\" (UniqueName: \"kubernetes.io/projected/ff0958b2-307f-49c2-8dc6-30d7958b2c91-kube-api-access-9trxh\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:08:40.710763 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.710535    4477 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ff0958b2-307f-49c2-8dc6-30d7958b2c91-cluster-tls-config\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:08:40.710763 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.710549    4477 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ff0958b2-307f-49c2-8dc6-30d7958b2c91-tls-assets\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:08:40.714140 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.714123    4477 scope.go:122] "RemoveContainer" containerID="fc2add1d153b0c4608d745a1d587f530ccf5a3c6bda7dd5d5508002d135d5ac4"
Oct 30 22:08:40.720945 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.720927    4477 scope.go:122] "RemoveContainer" containerID="94238bcd412582d3c1364e0a1d4c2dff953a9d36e46cc3a847f4d7d03c536222"
Oct 30 22:08:40.728713 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.728696    4477 scope.go:122] "RemoveContainer" containerID="b9989a6be10d37ab3c63475e7b987bf6bf195b0da0f26b807b467b486e8aa2c7"
Oct 30 22:08:40.740499 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.740483    4477 scope.go:122] "RemoveContainer" containerID="ed9c90533fb6507613ac9ee5aea4549c0abf8aabb46ebc6367c11781ac251772"
Oct 30 22:08:40.740641 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.740621    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 22:08:40.745630 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.745608    4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 22:08:40.748052 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.748019    4477 scope.go:122] "RemoveContainer" containerID="78603695f0a26f9b692426086e27124b57f7a1b424a4290084e8f6a51bee6a64"
Oct 30 22:08:40.748347 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:08:40.748326    4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78603695f0a26f9b692426086e27124b57f7a1b424a4290084e8f6a51bee6a64\": container with ID starting with 78603695f0a26f9b692426086e27124b57f7a1b424a4290084e8f6a51bee6a64 not found: ID does not exist" containerID="78603695f0a26f9b692426086e27124b57f7a1b424a4290084e8f6a51bee6a64"
Oct 30 22:08:40.748425 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.748364    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78603695f0a26f9b692426086e27124b57f7a1b424a4290084e8f6a51bee6a64"} err="failed to get container status \"78603695f0a26f9b692426086e27124b57f7a1b424a4290084e8f6a51bee6a64\": rpc error: code = NotFound desc = could not find container \"78603695f0a26f9b692426086e27124b57f7a1b424a4290084e8f6a51bee6a64\": container with ID starting with 78603695f0a26f9b692426086e27124b57f7a1b424a4290084e8f6a51bee6a64 not found: ID does not exist"
Oct 30 22:08:40.748425 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.748407    4477 scope.go:122] "RemoveContainer" containerID="c7c37e921f5dc77a58159775fc3f32410bfbe37ebd09609e8d3cc2eeb3b31a99"
Oct 30 22:08:40.748710 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:08:40.748692    4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7c37e921f5dc77a58159775fc3f32410bfbe37ebd09609e8d3cc2eeb3b31a99\": container with ID starting with c7c37e921f5dc77a58159775fc3f32410bfbe37ebd09609e8d3cc2eeb3b31a99 not found: ID does not exist" containerID="c7c37e921f5dc77a58159775fc3f32410bfbe37ebd09609e8d3cc2eeb3b31a99"
Oct 30 22:08:40.748780 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.748716    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7c37e921f5dc77a58159775fc3f32410bfbe37ebd09609e8d3cc2eeb3b31a99"} err="failed to get container status \"c7c37e921f5dc77a58159775fc3f32410bfbe37ebd09609e8d3cc2eeb3b31a99\": rpc error: code = NotFound desc = could not find container \"c7c37e921f5dc77a58159775fc3f32410bfbe37ebd09609e8d3cc2eeb3b31a99\": container with ID starting with c7c37e921f5dc77a58159775fc3f32410bfbe37ebd09609e8d3cc2eeb3b31a99 not found: ID does not exist"
Oct 30 22:08:40.748780 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.748732    4477 scope.go:122] "RemoveContainer" containerID="30da140289e5e2937a2dc54e4141d3650d66b3b64ab0e49a8e3de2fdb7213fac"
Oct 30 22:08:40.748968 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:08:40.748950    4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30da140289e5e2937a2dc54e4141d3650d66b3b64ab0e49a8e3de2fdb7213fac\": container with ID starting with 30da140289e5e2937a2dc54e4141d3650d66b3b64ab0e49a8e3de2fdb7213fac not found: ID does not exist" containerID="30da140289e5e2937a2dc54e4141d3650d66b3b64ab0e49a8e3de2fdb7213fac"
Oct 30 22:08:40.749035 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.748979    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30da140289e5e2937a2dc54e4141d3650d66b3b64ab0e49a8e3de2fdb7213fac"} err="failed to get container status \"30da140289e5e2937a2dc54e4141d3650d66b3b64ab0e49a8e3de2fdb7213fac\": rpc error: code = NotFound desc = could not find container \"30da140289e5e2937a2dc54e4141d3650d66b3b64ab0e49a8e3de2fdb7213fac\": container with ID starting with 30da140289e5e2937a2dc54e4141d3650d66b3b64ab0e49a8e3de2fdb7213fac not found: ID does not exist"
Oct 30 22:08:40.749035 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.749002    4477 scope.go:122] "RemoveContainer" containerID="fc2add1d153b0c4608d745a1d587f530ccf5a3c6bda7dd5d5508002d135d5ac4"
Oct 30 22:08:40.749295 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:08:40.749277    4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc2add1d153b0c4608d745a1d587f530ccf5a3c6bda7dd5d5508002d135d5ac4\": container with ID starting with fc2add1d153b0c4608d745a1d587f530ccf5a3c6bda7dd5d5508002d135d5ac4 not found: ID does not exist" containerID="fc2add1d153b0c4608d745a1d587f530ccf5a3c6bda7dd5d5508002d135d5ac4"
Oct 30 22:08:40.749331 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.749301    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc2add1d153b0c4608d745a1d587f530ccf5a3c6bda7dd5d5508002d135d5ac4"} err="failed to get container status \"fc2add1d153b0c4608d745a1d587f530ccf5a3c6bda7dd5d5508002d135d5ac4\": rpc error: code = NotFound desc = could not find container \"fc2add1d153b0c4608d745a1d587f530ccf5a3c6bda7dd5d5508002d135d5ac4\": container with ID starting with fc2add1d153b0c4608d745a1d587f530ccf5a3c6bda7dd5d5508002d135d5ac4 not found: ID does not exist"
Oct 30 22:08:40.749331 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.749318    4477 scope.go:122] "RemoveContainer" containerID="94238bcd412582d3c1364e0a1d4c2dff953a9d36e46cc3a847f4d7d03c536222"
Oct 30 22:08:40.749558 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:08:40.749541    4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94238bcd412582d3c1364e0a1d4c2dff953a9d36e46cc3a847f4d7d03c536222\": container with ID starting with 94238bcd412582d3c1364e0a1d4c2dff953a9d36e46cc3a847f4d7d03c536222 not found: ID does not exist" containerID="94238bcd412582d3c1364e0a1d4c2dff953a9d36e46cc3a847f4d7d03c536222"
Oct 30 22:08:40.749607 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.749565    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94238bcd412582d3c1364e0a1d4c2dff953a9d36e46cc3a847f4d7d03c536222"} err="failed to get container status \"94238bcd412582d3c1364e0a1d4c2dff953a9d36e46cc3a847f4d7d03c536222\": rpc error: code = NotFound desc = could not find container \"94238bcd412582d3c1364e0a1d4c2dff953a9d36e46cc3a847f4d7d03c536222\": container with ID starting with 94238bcd412582d3c1364e0a1d4c2dff953a9d36e46cc3a847f4d7d03c536222 not found: ID does not exist"
Oct 30 22:08:40.749607 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.749582    4477 scope.go:122] "RemoveContainer" containerID="b9989a6be10d37ab3c63475e7b987bf6bf195b0da0f26b807b467b486e8aa2c7"
Oct 30 22:08:40.749821 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:08:40.749802    4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9989a6be10d37ab3c63475e7b987bf6bf195b0da0f26b807b467b486e8aa2c7\": container with ID starting with b9989a6be10d37ab3c63475e7b987bf6bf195b0da0f26b807b467b486e8aa2c7 not found: ID does not exist" containerID="b9989a6be10d37ab3c63475e7b987bf6bf195b0da0f26b807b467b486e8aa2c7"
Oct 30 22:08:40.749821 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.749822    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9989a6be10d37ab3c63475e7b987bf6bf195b0da0f26b807b467b486e8aa2c7"} err="failed to get container status \"b9989a6be10d37ab3c63475e7b987bf6bf195b0da0f26b807b467b486e8aa2c7\": rpc error: code = NotFound desc = could not find container \"b9989a6be10d37ab3c63475e7b987bf6bf195b0da0f26b807b467b486e8aa2c7\": container with ID starting with b9989a6be10d37ab3c63475e7b987bf6bf195b0da0f26b807b467b486e8aa2c7 not found: ID does not exist"
Oct 30 22:08:40.749821 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.749835    4477 scope.go:122] "RemoveContainer" containerID="ed9c90533fb6507613ac9ee5aea4549c0abf8aabb46ebc6367c11781ac251772"
Oct 30 22:08:40.750088 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:08:40.750073    4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed9c90533fb6507613ac9ee5aea4549c0abf8aabb46ebc6367c11781ac251772\": container with ID starting with ed9c90533fb6507613ac9ee5aea4549c0abf8aabb46ebc6367c11781ac251772 not found: ID does not exist" containerID="ed9c90533fb6507613ac9ee5aea4549c0abf8aabb46ebc6367c11781ac251772"
Oct 30 22:08:40.750213 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.750088    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed9c90533fb6507613ac9ee5aea4549c0abf8aabb46ebc6367c11781ac251772"} err="failed to get container status \"ed9c90533fb6507613ac9ee5aea4549c0abf8aabb46ebc6367c11781ac251772\": rpc error: code = NotFound desc = could not find container \"ed9c90533fb6507613ac9ee5aea4549c0abf8aabb46ebc6367c11781ac251772\": container with ID starting with ed9c90533fb6507613ac9ee5aea4549c0abf8aabb46ebc6367c11781ac251772 not found: ID does not exist"
Oct 30 22:08:40.750213 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.750100    4477 scope.go:122] "RemoveContainer" containerID="78603695f0a26f9b692426086e27124b57f7a1b424a4290084e8f6a51bee6a64"
Oct 30 22:08:40.750328 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.750293    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78603695f0a26f9b692426086e27124b57f7a1b424a4290084e8f6a51bee6a64"} err="failed to get container status \"78603695f0a26f9b692426086e27124b57f7a1b424a4290084e8f6a51bee6a64\": rpc error: code = NotFound desc = could not find container \"78603695f0a26f9b692426086e27124b57f7a1b424a4290084e8f6a51bee6a64\": container with ID starting with 78603695f0a26f9b692426086e27124b57f7a1b424a4290084e8f6a51bee6a64 not found: ID does not exist"
Oct 30 22:08:40.750328 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.750312    4477 scope.go:122] "RemoveContainer" containerID="c7c37e921f5dc77a58159775fc3f32410bfbe37ebd09609e8d3cc2eeb3b31a99"
Oct 30 22:08:40.750664 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.750542    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7c37e921f5dc77a58159775fc3f32410bfbe37ebd09609e8d3cc2eeb3b31a99"} err="failed to get container status \"c7c37e921f5dc77a58159775fc3f32410bfbe37ebd09609e8d3cc2eeb3b31a99\": rpc error: code = NotFound desc = could not find container \"c7c37e921f5dc77a58159775fc3f32410bfbe37ebd09609e8d3cc2eeb3b31a99\": container with ID starting with c7c37e921f5dc77a58159775fc3f32410bfbe37ebd09609e8d3cc2eeb3b31a99 not found: ID does not exist"
Oct 30 22:08:40.750664 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.750561    4477 scope.go:122] "RemoveContainer" containerID="30da140289e5e2937a2dc54e4141d3650d66b3b64ab0e49a8e3de2fdb7213fac"
Oct 30 22:08:40.750858 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.750841    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30da140289e5e2937a2dc54e4141d3650d66b3b64ab0e49a8e3de2fdb7213fac"} err="failed to get container status \"30da140289e5e2937a2dc54e4141d3650d66b3b64ab0e49a8e3de2fdb7213fac\": rpc error: code = NotFound desc = could not find container \"30da140289e5e2937a2dc54e4141d3650d66b3b64ab0e49a8e3de2fdb7213fac\": container with ID starting with 30da140289e5e2937a2dc54e4141d3650d66b3b64ab0e49a8e3de2fdb7213fac not found: ID does not exist"
Oct 30 22:08:40.750858 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.750857    4477 scope.go:122] "RemoveContainer" containerID="fc2add1d153b0c4608d745a1d587f530ccf5a3c6bda7dd5d5508002d135d5ac4"
Oct 30 22:08:40.751157 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.751139    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc2add1d153b0c4608d745a1d587f530ccf5a3c6bda7dd5d5508002d135d5ac4"} err="failed to get container status \"fc2add1d153b0c4608d745a1d587f530ccf5a3c6bda7dd5d5508002d135d5ac4\": rpc error: code = NotFound desc = could not find container \"fc2add1d153b0c4608d745a1d587f530ccf5a3c6bda7dd5d5508002d135d5ac4\": container with ID starting with fc2add1d153b0c4608d745a1d587f530ccf5a3c6bda7dd5d5508002d135d5ac4 not found: ID does not exist"
Oct 30 22:08:40.751263 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.751156    4477 scope.go:122] "RemoveContainer" containerID="94238bcd412582d3c1364e0a1d4c2dff953a9d36e46cc3a847f4d7d03c536222"
Oct 30 22:08:40.751519 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.751495    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94238bcd412582d3c1364e0a1d4c2dff953a9d36e46cc3a847f4d7d03c536222"} err="failed to get container status \"94238bcd412582d3c1364e0a1d4c2dff953a9d36e46cc3a847f4d7d03c536222\": rpc error: code = NotFound desc = could not find container \"94238bcd412582d3c1364e0a1d4c2dff953a9d36e46cc3a847f4d7d03c536222\": container with ID starting with 94238bcd412582d3c1364e0a1d4c2dff953a9d36e46cc3a847f4d7d03c536222 not found: ID does not exist"
Oct 30 22:08:40.751620 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.751527    4477 scope.go:122] "RemoveContainer" containerID="b9989a6be10d37ab3c63475e7b987bf6bf195b0da0f26b807b467b486e8aa2c7"
Oct 30 22:08:40.751809 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.751782    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9989a6be10d37ab3c63475e7b987bf6bf195b0da0f26b807b467b486e8aa2c7"} err="failed to get container status \"b9989a6be10d37ab3c63475e7b987bf6bf195b0da0f26b807b467b486e8aa2c7\": rpc error: code = NotFound desc = could not find container \"b9989a6be10d37ab3c63475e7b987bf6bf195b0da0f26b807b467b486e8aa2c7\": container with ID starting with b9989a6be10d37ab3c63475e7b987bf6bf195b0da0f26b807b467b486e8aa2c7 not found: ID does not exist"
Oct 30 22:08:40.751878 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.751810    4477 scope.go:122] "RemoveContainer" containerID="ed9c90533fb6507613ac9ee5aea4549c0abf8aabb46ebc6367c11781ac251772"
Oct 30 22:08:40.752528 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.752484    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed9c90533fb6507613ac9ee5aea4549c0abf8aabb46ebc6367c11781ac251772"} err="failed to get container status \"ed9c90533fb6507613ac9ee5aea4549c0abf8aabb46ebc6367c11781ac251772\": rpc error: code = NotFound desc = could not find container \"ed9c90533fb6507613ac9ee5aea4549c0abf8aabb46ebc6367c11781ac251772\": container with ID starting with ed9c90533fb6507613ac9ee5aea4549c0abf8aabb46ebc6367c11781ac251772 not found: ID does not exist"
Oct 30 22:08:40.777511 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.777475    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 22:08:40.777986 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.777964    4477 cpu_manager.go:414] "containerMap: removing container" podUID="ff0958b2-307f-49c2-8dc6-30d7958b2c91" containerName="kube-rbac-proxy"
Oct 30 22:08:40.778168 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.778145    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff0958b2-307f-49c2-8dc6-30d7958b2c91" containerName="kube-rbac-proxy"
Oct 30 22:08:40.778285 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.778178    4477 cpu_manager.go:414] "containerMap: removing container" podUID="ff0958b2-307f-49c2-8dc6-30d7958b2c91" containerName="kube-rbac-proxy-metric"
Oct 30 22:08:40.778285 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.778186    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff0958b2-307f-49c2-8dc6-30d7958b2c91" containerName="kube-rbac-proxy-metric"
Oct 30 22:08:40.778285 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.778214    4477 cpu_manager.go:414] "containerMap: removing container" podUID="ff0958b2-307f-49c2-8dc6-30d7958b2c91" containerName="init-config-reloader"
Oct 30 22:08:40.778285 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.778221    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff0958b2-307f-49c2-8dc6-30d7958b2c91" containerName="init-config-reloader"
Oct 30 22:08:40.778285 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.778234    4477 cpu_manager.go:414] "containerMap: removing container" podUID="ff0958b2-307f-49c2-8dc6-30d7958b2c91" containerName="alertmanager"
Oct 30 22:08:40.778285 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.778241    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff0958b2-307f-49c2-8dc6-30d7958b2c91" containerName="alertmanager"
Oct 30 22:08:40.778285 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.778260    4477 cpu_manager.go:414] "containerMap: removing container" podUID="ff0958b2-307f-49c2-8dc6-30d7958b2c91" containerName="kube-rbac-proxy-web"
Oct 30 22:08:40.778285 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.778267    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff0958b2-307f-49c2-8dc6-30d7958b2c91" containerName="kube-rbac-proxy-web"
Oct 30 22:08:40.778285 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.778279    4477 cpu_manager.go:414] "containerMap: removing container" podUID="ff0958b2-307f-49c2-8dc6-30d7958b2c91" containerName="prom-label-proxy"
Oct 30 22:08:40.778285 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.778285    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff0958b2-307f-49c2-8dc6-30d7958b2c91" containerName="prom-label-proxy"
Oct 30 22:08:40.778629 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.778296    4477 cpu_manager.go:414] "containerMap: removing container" podUID="ff0958b2-307f-49c2-8dc6-30d7958b2c91" containerName="config-reloader"
Oct 30 22:08:40.778629 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.778303    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff0958b2-307f-49c2-8dc6-30d7958b2c91" containerName="config-reloader"
Oct 30 22:08:40.778629 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.778408    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff0958b2-307f-49c2-8dc6-30d7958b2c91" containerName="kube-rbac-proxy"
Oct 30 22:08:40.778629 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.778423    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff0958b2-307f-49c2-8dc6-30d7958b2c91" containerName="kube-rbac-proxy-metric"
Oct 30 22:08:40.778629 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.778435    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff0958b2-307f-49c2-8dc6-30d7958b2c91" containerName="alertmanager"
Oct 30 22:08:40.778629 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.778448    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff0958b2-307f-49c2-8dc6-30d7958b2c91" containerName="kube-rbac-proxy-web"
Oct 30 22:08:40.778629 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.778456    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff0958b2-307f-49c2-8dc6-30d7958b2c91" containerName="prom-label-proxy"
Oct 30 22:08:40.778629 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.778462    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff0958b2-307f-49c2-8dc6-30d7958b2c91" containerName="config-reloader"
Oct 30 22:08:40.783087 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.783069    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 22:08:40.798605 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.798581    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Oct 30 22:08:40.800181 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.800164    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Oct 30 22:08:40.800292 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.800278    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-jtfxw\""
Oct 30 22:08:40.800366 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.800354    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Oct 30 22:08:40.800454 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.800444    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Oct 30 22:08:40.800527 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.800516    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Oct 30 22:08:40.800583 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.800573    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Oct 30 22:08:40.800632 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.800627    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Oct 30 22:08:40.800698 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.800688    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Oct 30 22:08:40.807512 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.807494    4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Oct 30 22:08:40.813774 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.813752    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 22:08:40.911997 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.911959    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4c435879-6dad-49c3-a9fa-fc1d7390d69e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"4c435879-6dad-49c3-a9fa-fc1d7390d69e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 22:08:40.912188 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.912033    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4c435879-6dad-49c3-a9fa-fc1d7390d69e-config-out\") pod \"alertmanager-main-0\" (UID: \"4c435879-6dad-49c3-a9fa-fc1d7390d69e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 22:08:40.912188 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.912097    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4c435879-6dad-49c3-a9fa-fc1d7390d69e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"4c435879-6dad-49c3-a9fa-fc1d7390d69e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 22:08:40.912188 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.912124    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4c435879-6dad-49c3-a9fa-fc1d7390d69e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"4c435879-6dad-49c3-a9fa-fc1d7390d69e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 22:08:40.912188 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.912152    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4c435879-6dad-49c3-a9fa-fc1d7390d69e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"4c435879-6dad-49c3-a9fa-fc1d7390d69e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 22:08:40.912360 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.912281    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4c435879-6dad-49c3-a9fa-fc1d7390d69e-config-volume\") pod \"alertmanager-main-0\" (UID: \"4c435879-6dad-49c3-a9fa-fc1d7390d69e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 22:08:40.912360 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.912329    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4c435879-6dad-49c3-a9fa-fc1d7390d69e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"4c435879-6dad-49c3-a9fa-fc1d7390d69e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 22:08:40.912360 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.912354    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4c435879-6dad-49c3-a9fa-fc1d7390d69e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"4c435879-6dad-49c3-a9fa-fc1d7390d69e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 22:08:40.912459 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.912381    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4c435879-6dad-49c3-a9fa-fc1d7390d69e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"4c435879-6dad-49c3-a9fa-fc1d7390d69e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 22:08:40.912459 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.912410    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6d5s5\" (UniqueName: \"kubernetes.io/projected/4c435879-6dad-49c3-a9fa-fc1d7390d69e-kube-api-access-6d5s5\") pod \"alertmanager-main-0\" (UID: \"4c435879-6dad-49c3-a9fa-fc1d7390d69e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 22:08:40.912459 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.912430    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4c435879-6dad-49c3-a9fa-fc1d7390d69e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"4c435879-6dad-49c3-a9fa-fc1d7390d69e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 22:08:40.912561 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.912460    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4c435879-6dad-49c3-a9fa-fc1d7390d69e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"4c435879-6dad-49c3-a9fa-fc1d7390d69e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 22:08:40.912561 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:40.912481    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4c435879-6dad-49c3-a9fa-fc1d7390d69e-web-config\") pod \"alertmanager-main-0\" (UID: \"4c435879-6dad-49c3-a9fa-fc1d7390d69e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 22:08:41.013218 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:41.013189    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6d5s5\" (UniqueName: \"kubernetes.io/projected/4c435879-6dad-49c3-a9fa-fc1d7390d69e-kube-api-access-6d5s5\") pod \"alertmanager-main-0\" (UID: \"4c435879-6dad-49c3-a9fa-fc1d7390d69e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 22:08:41.013375 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:41.013227    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4c435879-6dad-49c3-a9fa-fc1d7390d69e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"4c435879-6dad-49c3-a9fa-fc1d7390d69e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 22:08:41.013375 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:41.013244    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4c435879-6dad-49c3-a9fa-fc1d7390d69e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"4c435879-6dad-49c3-a9fa-fc1d7390d69e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 22:08:41.013375 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:41.013263    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4c435879-6dad-49c3-a9fa-fc1d7390d69e-web-config\") pod \"alertmanager-main-0\" (UID: \"4c435879-6dad-49c3-a9fa-fc1d7390d69e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 22:08:41.013549 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:41.013395    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4c435879-6dad-49c3-a9fa-fc1d7390d69e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"4c435879-6dad-49c3-a9fa-fc1d7390d69e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 22:08:41.013549 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:41.013469    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4c435879-6dad-49c3-a9fa-fc1d7390d69e-config-out\") pod \"alertmanager-main-0\" (UID: \"4c435879-6dad-49c3-a9fa-fc1d7390d69e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 22:08:41.013549 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:41.013498    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4c435879-6dad-49c3-a9fa-fc1d7390d69e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"4c435879-6dad-49c3-a9fa-fc1d7390d69e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 22:08:41.013549 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:41.013522    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4c435879-6dad-49c3-a9fa-fc1d7390d69e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"4c435879-6dad-49c3-a9fa-fc1d7390d69e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 22:08:41.013766 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:41.013669    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4c435879-6dad-49c3-a9fa-fc1d7390d69e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"4c435879-6dad-49c3-a9fa-fc1d7390d69e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 22:08:41.013766 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:41.013735    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4c435879-6dad-49c3-a9fa-fc1d7390d69e-config-volume\") pod \"alertmanager-main-0\" (UID: \"4c435879-6dad-49c3-a9fa-fc1d7390d69e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 22:08:41.013874 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:41.013788    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4c435879-6dad-49c3-a9fa-fc1d7390d69e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"4c435879-6dad-49c3-a9fa-fc1d7390d69e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 22:08:41.013874 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:41.013820    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4c435879-6dad-49c3-a9fa-fc1d7390d69e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"4c435879-6dad-49c3-a9fa-fc1d7390d69e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 22:08:41.013874 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:41.013858    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4c435879-6dad-49c3-a9fa-fc1d7390d69e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"4c435879-6dad-49c3-a9fa-fc1d7390d69e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 22:08:41.014692 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:41.014668    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4c435879-6dad-49c3-a9fa-fc1d7390d69e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"4c435879-6dad-49c3-a9fa-fc1d7390d69e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 22:08:41.016448 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:41.016156    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4c435879-6dad-49c3-a9fa-fc1d7390d69e-config-out\") pod \"alertmanager-main-0\" (UID: \"4c435879-6dad-49c3-a9fa-fc1d7390d69e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 22:08:41.016448 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:41.016282    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4c435879-6dad-49c3-a9fa-fc1d7390d69e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"4c435879-6dad-49c3-a9fa-fc1d7390d69e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 22:08:41.016448 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:41.016279    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4c435879-6dad-49c3-a9fa-fc1d7390d69e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"4c435879-6dad-49c3-a9fa-fc1d7390d69e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 22:08:41.016448 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:41.016384    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4c435879-6dad-49c3-a9fa-fc1d7390d69e-web-config\") pod \"alertmanager-main-0\" (UID: \"4c435879-6dad-49c3-a9fa-fc1d7390d69e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 22:08:41.016448 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:41.016396    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4c435879-6dad-49c3-a9fa-fc1d7390d69e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"4c435879-6dad-49c3-a9fa-fc1d7390d69e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 22:08:41.016448 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:41.016422    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4c435879-6dad-49c3-a9fa-fc1d7390d69e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"4c435879-6dad-49c3-a9fa-fc1d7390d69e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 22:08:41.016718 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:41.016503    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4c435879-6dad-49c3-a9fa-fc1d7390d69e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"4c435879-6dad-49c3-a9fa-fc1d7390d69e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 22:08:41.016858 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:41.016837    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4c435879-6dad-49c3-a9fa-fc1d7390d69e-config-volume\") pod \"alertmanager-main-0\" (UID: \"4c435879-6dad-49c3-a9fa-fc1d7390d69e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 22:08:41.016942 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:41.016924    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4c435879-6dad-49c3-a9fa-fc1d7390d69e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"4c435879-6dad-49c3-a9fa-fc1d7390d69e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 22:08:41.017263 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:41.017247    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4c435879-6dad-49c3-a9fa-fc1d7390d69e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"4c435879-6dad-49c3-a9fa-fc1d7390d69e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 22:08:41.018163 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:41.018147    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4c435879-6dad-49c3-a9fa-fc1d7390d69e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"4c435879-6dad-49c3-a9fa-fc1d7390d69e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 22:08:41.031410 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:41.031390    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6d5s5\" (UniqueName: \"kubernetes.io/projected/4c435879-6dad-49c3-a9fa-fc1d7390d69e-kube-api-access-6d5s5\") pod \"alertmanager-main-0\" (UID: \"4c435879-6dad-49c3-a9fa-fc1d7390d69e\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 22:08:41.104259 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:41.103646    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 22:08:41.454861 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:41.454791    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 22:08:41.696735 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:41.696699    4477 generic.go:356] "Generic (PLEG): container finished" podID="4c435879-6dad-49c3-a9fa-fc1d7390d69e" containerID="5c97baf1d8f49559734911429d3d7f1a7403d461daf139ba9a8a3a5b9e81487d" exitCode=0
Oct 30 22:08:41.696911 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:41.696752    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4c435879-6dad-49c3-a9fa-fc1d7390d69e","Type":"ContainerDied","Data":"5c97baf1d8f49559734911429d3d7f1a7403d461daf139ba9a8a3a5b9e81487d"}
Oct 30 22:08:41.696911 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:41.696814    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4c435879-6dad-49c3-a9fa-fc1d7390d69e","Type":"ContainerStarted","Data":"bf24d05905ba6914cbde6a3f7c6b739706f4153b4f8f3c444c979c9028c51db7"}
Oct 30 22:08:42.193637 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:42.193606    4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff0958b2-307f-49c2-8dc6-30d7958b2c91" path="/var/lib/kubelet/pods/ff0958b2-307f-49c2-8dc6-30d7958b2c91/volumes"
Oct 30 22:08:42.703654 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:42.703624    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4c435879-6dad-49c3-a9fa-fc1d7390d69e","Type":"ContainerStarted","Data":"63805926ced056289a697889d668b4d25a59da6686cc5832e3c3c59c74a1582d"}
Oct 30 22:08:42.704013 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:42.703688    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4c435879-6dad-49c3-a9fa-fc1d7390d69e","Type":"ContainerStarted","Data":"a28ac916d0e8e7a2b37f0f134ab931bded820b2276b86b6dd1a7f611f5396e4d"}
Oct 30 22:08:42.704013 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:42.703715    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4c435879-6dad-49c3-a9fa-fc1d7390d69e","Type":"ContainerStarted","Data":"f772f80860f140cc25d66adfdb6c79ec9c10d7678c867f46c81de1ab273cd2ab"}
Oct 30 22:08:42.704013 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:42.703740    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4c435879-6dad-49c3-a9fa-fc1d7390d69e","Type":"ContainerStarted","Data":"457e9a9d28f58e31bc600de45498136f48736d945f857b59b38ab648e00d3d08"}
Oct 30 22:08:42.704013 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:42.703777    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4c435879-6dad-49c3-a9fa-fc1d7390d69e","Type":"ContainerStarted","Data":"baf98b487459a7af97000d208ecc3f6df5567621ca7d14e359b35d5b5d620dec"}
Oct 30 22:08:42.704013 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:42.703827    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4c435879-6dad-49c3-a9fa-fc1d7390d69e","Type":"ContainerStarted","Data":"72dc45865a65878942f84b11c6ca7a12e9da4bd044f11663da1cdd7bc7f92ef7"}
Oct 30 22:08:42.772019 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:42.771976    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.757902983 podStartE2EDuration="2.771961602s" totalImagesPullingTime="0s" totalInitContainerRuntime="14.058619ms" isStatefulPod=false podCreationTimestamp="2026-10-30 22:08:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:08:42.760479655 +0000 UTC m=+106.686405865" watchObservedRunningTime="2026-10-30 22:08:42.771961602 +0000 UTC m=+106.697887807"
Oct 30 22:08:43.806451 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:43.806420    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 22:08:43.806886 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:43.806735    4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f904edd0-10dc-4958-8d9b-9e012c1319db" containerName="kube-rbac-proxy" containerID="cri-o://0fe9c11ae206823bc898cf13163d98146cb190eb2952e4e250b30ed2b8b501ba" gracePeriod=600
Oct 30 22:08:43.806886 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:43.806780    4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f904edd0-10dc-4958-8d9b-9e012c1319db" containerName="kube-rbac-proxy-web" containerID="cri-o://6cd67108b4e7516c87b9a3cadbfd54edd7d436841f49ce876fc03d7d718ca2ec" gracePeriod=600
Oct 30 22:08:43.807008 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:43.806880    4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f904edd0-10dc-4958-8d9b-9e012c1319db" containerName="config-reloader" containerID="cri-o://c0c298ca2a98a892f23adecaddc83575f89fb6ed37cd952300aa86954ca586fb" gracePeriod=600
Oct 30 22:08:43.807008 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:43.806932    4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f904edd0-10dc-4958-8d9b-9e012c1319db" containerName="kube-rbac-proxy-thanos" containerID="cri-o://e59f8a472d1f4bf287761c5eccf03ceed9f5b1b9243e84a9dd2cf9faa9c9801a" gracePeriod=600
Oct 30 22:08:43.807008 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:43.806964    4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f904edd0-10dc-4958-8d9b-9e012c1319db" containerName="thanos-sidecar" containerID="cri-o://1b3367b9f00d774fd20bfb6349716fbc15f118a8d18bbfc4d0fd9af8c72c63f7" gracePeriod=600
Oct 30 22:08:43.807189 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:43.806723    4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="f904edd0-10dc-4958-8d9b-9e012c1319db" containerName="prometheus" containerID="cri-o://e3cd27a71f050df532a08814a4280c80556187adb7c8d22fa82cd343369a30e0" gracePeriod=600
Oct 30 22:08:44.038541 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.038518    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:08:44.048657 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.048635    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f904edd0-10dc-4958-8d9b-9e012c1319db-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f904edd0-10dc-4958-8d9b-9e012c1319db-secret-prometheus-k8s-tls\") pod \"f904edd0-10dc-4958-8d9b-9e012c1319db\" (UID: \"f904edd0-10dc-4958-8d9b-9e012c1319db\") "
Oct 30 22:08:44.048757 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.048677    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f904edd0-10dc-4958-8d9b-9e012c1319db-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f904edd0-10dc-4958-8d9b-9e012c1319db-prometheus-k8s-rulefiles-2\") pod \"f904edd0-10dc-4958-8d9b-9e012c1319db\" (UID: \"f904edd0-10dc-4958-8d9b-9e012c1319db\") "
Oct 30 22:08:44.048757 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.048722    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f904edd0-10dc-4958-8d9b-9e012c1319db-web-config\" (UniqueName: \"kubernetes.io/secret/f904edd0-10dc-4958-8d9b-9e012c1319db-web-config\") pod \"f904edd0-10dc-4958-8d9b-9e012c1319db\" (UID: \"f904edd0-10dc-4958-8d9b-9e012c1319db\") "
Oct 30 22:08:44.048757 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.048744    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f904edd0-10dc-4958-8d9b-9e012c1319db-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f904edd0-10dc-4958-8d9b-9e012c1319db-prometheus-k8s-rulefiles-1\") pod \"f904edd0-10dc-4958-8d9b-9e012c1319db\" (UID: \"f904edd0-10dc-4958-8d9b-9e012c1319db\") "
Oct 30 22:08:44.048916 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.048769    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f904edd0-10dc-4958-8d9b-9e012c1319db-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f904edd0-10dc-4958-8d9b-9e012c1319db-prometheus-trusted-ca-bundle\") pod \"f904edd0-10dc-4958-8d9b-9e012c1319db\" (UID: \"f904edd0-10dc-4958-8d9b-9e012c1319db\") "
Oct 30 22:08:44.048969 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.048923    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f904edd0-10dc-4958-8d9b-9e012c1319db-config-out\" (UniqueName: \"kubernetes.io/empty-dir/f904edd0-10dc-4958-8d9b-9e012c1319db-config-out\") pod \"f904edd0-10dc-4958-8d9b-9e012c1319db\" (UID: \"f904edd0-10dc-4958-8d9b-9e012c1319db\") "
Oct 30 22:08:44.049029 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.048977    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f904edd0-10dc-4958-8d9b-9e012c1319db-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f904edd0-10dc-4958-8d9b-9e012c1319db-secret-grpc-tls\") pod \"f904edd0-10dc-4958-8d9b-9e012c1319db\" (UID: \"f904edd0-10dc-4958-8d9b-9e012c1319db\") "
Oct 30 22:08:44.049029 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.049012    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f904edd0-10dc-4958-8d9b-9e012c1319db-tls-assets\" (UniqueName: \"kubernetes.io/projected/f904edd0-10dc-4958-8d9b-9e012c1319db-tls-assets\") pod \"f904edd0-10dc-4958-8d9b-9e012c1319db\" (UID: \"f904edd0-10dc-4958-8d9b-9e012c1319db\") "
Oct 30 22:08:44.049158 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.049057    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f904edd0-10dc-4958-8d9b-9e012c1319db-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f904edd0-10dc-4958-8d9b-9e012c1319db-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"f904edd0-10dc-4958-8d9b-9e012c1319db\" (UID: \"f904edd0-10dc-4958-8d9b-9e012c1319db\") "
Oct 30 22:08:44.049158 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.049092    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f904edd0-10dc-4958-8d9b-9e012c1319db-prometheus-k8s-rulefiles-2" pod "f904edd0-10dc-4958-8d9b-9e012c1319db" (UID: "f904edd0-10dc-4958-8d9b-9e012c1319db"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:08:44.049158 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.049102    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f904edd0-10dc-4958-8d9b-9e012c1319db-prometheus-k8s-rulefiles-1" pod "f904edd0-10dc-4958-8d9b-9e012c1319db" (UID: "f904edd0-10dc-4958-8d9b-9e012c1319db"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:08:44.049158 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.049098    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f904edd0-10dc-4958-8d9b-9e012c1319db-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f904edd0-10dc-4958-8d9b-9e012c1319db-configmap-metrics-client-ca\") pod \"f904edd0-10dc-4958-8d9b-9e012c1319db\" (UID: \"f904edd0-10dc-4958-8d9b-9e012c1319db\") "
Oct 30 22:08:44.049158 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.049147    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f904edd0-10dc-4958-8d9b-9e012c1319db-prometheus-trusted-ca-bundle" pod "f904edd0-10dc-4958-8d9b-9e012c1319db" (UID: "f904edd0-10dc-4958-8d9b-9e012c1319db"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:08:44.049158 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.049156    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f904edd0-10dc-4958-8d9b-9e012c1319db-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f904edd0-10dc-4958-8d9b-9e012c1319db-secret-kube-rbac-proxy\") pod \"f904edd0-10dc-4958-8d9b-9e012c1319db\" (UID: \"f904edd0-10dc-4958-8d9b-9e012c1319db\") "
Oct 30 22:08:44.049453 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.049190    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f904edd0-10dc-4958-8d9b-9e012c1319db-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f904edd0-10dc-4958-8d9b-9e012c1319db-configmap-serving-certs-ca-bundle\") pod \"f904edd0-10dc-4958-8d9b-9e012c1319db\" (UID: \"f904edd0-10dc-4958-8d9b-9e012c1319db\") "
Oct 30 22:08:44.049453 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.049226    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f904edd0-10dc-4958-8d9b-9e012c1319db-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f904edd0-10dc-4958-8d9b-9e012c1319db-prometheus-k8s-db\") pod \"f904edd0-10dc-4958-8d9b-9e012c1319db\" (UID: \"f904edd0-10dc-4958-8d9b-9e012c1319db\") "
Oct 30 22:08:44.049453 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.049325    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f904edd0-10dc-4958-8d9b-9e012c1319db-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f904edd0-10dc-4958-8d9b-9e012c1319db-configmap-kubelet-serving-ca-bundle\") pod \"f904edd0-10dc-4958-8d9b-9e012c1319db\" (UID: \"f904edd0-10dc-4958-8d9b-9e012c1319db\") "
Oct 30 22:08:44.049453 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.049376    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f904edd0-10dc-4958-8d9b-9e012c1319db-kube-api-access-9h2cn\" (UniqueName: \"kubernetes.io/projected/f904edd0-10dc-4958-8d9b-9e012c1319db-kube-api-access-9h2cn\") pod \"f904edd0-10dc-4958-8d9b-9e012c1319db\" (UID: \"f904edd0-10dc-4958-8d9b-9e012c1319db\") "
Oct 30 22:08:44.049453 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.049424    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f904edd0-10dc-4958-8d9b-9e012c1319db-configmap-metrics-client-ca" pod "f904edd0-10dc-4958-8d9b-9e012c1319db" (UID: "f904edd0-10dc-4958-8d9b-9e012c1319db"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:08:44.049453 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.049433    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f904edd0-10dc-4958-8d9b-9e012c1319db-config\" (UniqueName: \"kubernetes.io/secret/f904edd0-10dc-4958-8d9b-9e012c1319db-config\") pod \"f904edd0-10dc-4958-8d9b-9e012c1319db\" (UID: \"f904edd0-10dc-4958-8d9b-9e012c1319db\") "
Oct 30 22:08:44.049751 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.049478    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f904edd0-10dc-4958-8d9b-9e012c1319db-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f904edd0-10dc-4958-8d9b-9e012c1319db-thanos-prometheus-http-client-file\") pod \"f904edd0-10dc-4958-8d9b-9e012c1319db\" (UID: \"f904edd0-10dc-4958-8d9b-9e012c1319db\") "
Oct 30 22:08:44.049751 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.049525    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f904edd0-10dc-4958-8d9b-9e012c1319db-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f904edd0-10dc-4958-8d9b-9e012c1319db-prometheus-k8s-rulefiles-0\") pod \"f904edd0-10dc-4958-8d9b-9e012c1319db\" (UID: \"f904edd0-10dc-4958-8d9b-9e012c1319db\") "
Oct 30 22:08:44.049751 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.049555    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f904edd0-10dc-4958-8d9b-9e012c1319db-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f904edd0-10dc-4958-8d9b-9e012c1319db-secret-metrics-client-certs\") pod \"f904edd0-10dc-4958-8d9b-9e012c1319db\" (UID: \"f904edd0-10dc-4958-8d9b-9e012c1319db\") "
Oct 30 22:08:44.050678 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.050425    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f904edd0-10dc-4958-8d9b-9e012c1319db-configmap-kubelet-serving-ca-bundle" pod "f904edd0-10dc-4958-8d9b-9e012c1319db" (UID: "f904edd0-10dc-4958-8d9b-9e012c1319db"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:08:44.051076 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.051017    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f904edd0-10dc-4958-8d9b-9e012c1319db-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f904edd0-10dc-4958-8d9b-9e012c1319db-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"f904edd0-10dc-4958-8d9b-9e012c1319db\" (UID: \"f904edd0-10dc-4958-8d9b-9e012c1319db\") "
Oct 30 22:08:44.051076 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.051065    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f904edd0-10dc-4958-8d9b-9e012c1319db-prometheus-k8s-db" pod "f904edd0-10dc-4958-8d9b-9e012c1319db" (UID: "f904edd0-10dc-4958-8d9b-9e012c1319db"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:08:44.051674 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.051317    4477 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f904edd0-10dc-4958-8d9b-9e012c1319db-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:08:44.051674 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.051342    4477 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f904edd0-10dc-4958-8d9b-9e012c1319db-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:08:44.051674 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.051359    4477 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f904edd0-10dc-4958-8d9b-9e012c1319db-prometheus-trusted-ca-bundle\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:08:44.051674 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.051374    4477 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f904edd0-10dc-4958-8d9b-9e012c1319db-configmap-metrics-client-ca\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:08:44.051674 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.051389    4477 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f904edd0-10dc-4958-8d9b-9e012c1319db-prometheus-k8s-db\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:08:44.051674 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.051405    4477 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f904edd0-10dc-4958-8d9b-9e012c1319db-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:08:44.051674 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.051431    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f904edd0-10dc-4958-8d9b-9e012c1319db-configmap-serving-certs-ca-bundle" pod "f904edd0-10dc-4958-8d9b-9e012c1319db" (UID: "f904edd0-10dc-4958-8d9b-9e012c1319db"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:08:44.052328 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.052300    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f904edd0-10dc-4958-8d9b-9e012c1319db-config" pod "f904edd0-10dc-4958-8d9b-9e012c1319db" (UID: "f904edd0-10dc-4958-8d9b-9e012c1319db"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:08:44.052840 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.052812    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f904edd0-10dc-4958-8d9b-9e012c1319db-secret-prometheus-k8s-kube-rbac-proxy-web" pod "f904edd0-10dc-4958-8d9b-9e012c1319db" (UID: "f904edd0-10dc-4958-8d9b-9e012c1319db"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:08:44.052942 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.052923    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f904edd0-10dc-4958-8d9b-9e012c1319db-secret-kube-rbac-proxy" pod "f904edd0-10dc-4958-8d9b-9e012c1319db" (UID: "f904edd0-10dc-4958-8d9b-9e012c1319db"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:08:44.053333 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.053279    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f904edd0-10dc-4958-8d9b-9e012c1319db-prometheus-k8s-rulefiles-0" pod "f904edd0-10dc-4958-8d9b-9e012c1319db" (UID: "f904edd0-10dc-4958-8d9b-9e012c1319db"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:08:44.053520 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.053451    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f904edd0-10dc-4958-8d9b-9e012c1319db-secret-prometheus-k8s-tls" pod "f904edd0-10dc-4958-8d9b-9e012c1319db" (UID: "f904edd0-10dc-4958-8d9b-9e012c1319db"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:08:44.053520 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.053473    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f904edd0-10dc-4958-8d9b-9e012c1319db-config-out" pod "f904edd0-10dc-4958-8d9b-9e012c1319db" (UID: "f904edd0-10dc-4958-8d9b-9e012c1319db"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:08:44.053652 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.053526    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f904edd0-10dc-4958-8d9b-9e012c1319db-tls-assets" pod "f904edd0-10dc-4958-8d9b-9e012c1319db" (UID: "f904edd0-10dc-4958-8d9b-9e012c1319db"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:08:44.053652 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.053575    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f904edd0-10dc-4958-8d9b-9e012c1319db-thanos-prometheus-http-client-file" pod "f904edd0-10dc-4958-8d9b-9e012c1319db" (UID: "f904edd0-10dc-4958-8d9b-9e012c1319db"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:08:44.053763 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.053652    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f904edd0-10dc-4958-8d9b-9e012c1319db-secret-grpc-tls" pod "f904edd0-10dc-4958-8d9b-9e012c1319db" (UID: "f904edd0-10dc-4958-8d9b-9e012c1319db"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:08:44.054553 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.054535    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f904edd0-10dc-4958-8d9b-9e012c1319db-kube-api-access-9h2cn" pod "f904edd0-10dc-4958-8d9b-9e012c1319db" (UID: "f904edd0-10dc-4958-8d9b-9e012c1319db"). InnerVolumeSpecName "kube-api-access-9h2cn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:08:44.054898 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.054880    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f904edd0-10dc-4958-8d9b-9e012c1319db-secret-metrics-client-certs" pod "f904edd0-10dc-4958-8d9b-9e012c1319db" (UID: "f904edd0-10dc-4958-8d9b-9e012c1319db"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:08:44.055026 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.055008    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f904edd0-10dc-4958-8d9b-9e012c1319db-secret-prometheus-k8s-thanos-sidecar-tls" pod "f904edd0-10dc-4958-8d9b-9e012c1319db" (UID: "f904edd0-10dc-4958-8d9b-9e012c1319db"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:08:44.063527 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.063481    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f904edd0-10dc-4958-8d9b-9e012c1319db-web-config" pod "f904edd0-10dc-4958-8d9b-9e012c1319db" (UID: "f904edd0-10dc-4958-8d9b-9e012c1319db"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:08:44.152653 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.152629    4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9h2cn\" (UniqueName: \"kubernetes.io/projected/f904edd0-10dc-4958-8d9b-9e012c1319db-kube-api-access-9h2cn\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:08:44.152653 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.152652    4477 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/f904edd0-10dc-4958-8d9b-9e012c1319db-config\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:08:44.152813 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.152668    4477 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f904edd0-10dc-4958-8d9b-9e012c1319db-thanos-prometheus-http-client-file\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:08:44.152813 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.152679    4477 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f904edd0-10dc-4958-8d9b-9e012c1319db-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:08:44.152813 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.152689    4477 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f904edd0-10dc-4958-8d9b-9e012c1319db-secret-metrics-client-certs\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:08:44.152813 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.152698    4477 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f904edd0-10dc-4958-8d9b-9e012c1319db-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:08:44.152813 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.152707    4477 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f904edd0-10dc-4958-8d9b-9e012c1319db-secret-prometheus-k8s-tls\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:08:44.152813 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.152715    4477 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f904edd0-10dc-4958-8d9b-9e012c1319db-web-config\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:08:44.152813 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.152722    4477 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f904edd0-10dc-4958-8d9b-9e012c1319db-config-out\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:08:44.152813 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.152730    4477 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f904edd0-10dc-4958-8d9b-9e012c1319db-secret-grpc-tls\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:08:44.152813 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.152737    4477 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f904edd0-10dc-4958-8d9b-9e012c1319db-tls-assets\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:08:44.152813 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.152745    4477 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f904edd0-10dc-4958-8d9b-9e012c1319db-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:08:44.152813 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.152753    4477 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f904edd0-10dc-4958-8d9b-9e012c1319db-secret-kube-rbac-proxy\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:08:44.152813 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.152761    4477 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f904edd0-10dc-4958-8d9b-9e012c1319db-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:08:44.713213 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.713178    4477 generic.go:356] "Generic (PLEG): container finished" podID="f904edd0-10dc-4958-8d9b-9e012c1319db" containerID="e59f8a472d1f4bf287761c5eccf03ceed9f5b1b9243e84a9dd2cf9faa9c9801a" exitCode=0
Oct 30 22:08:44.713213 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.713204    4477 generic.go:356] "Generic (PLEG): container finished" podID="f904edd0-10dc-4958-8d9b-9e012c1319db" containerID="0fe9c11ae206823bc898cf13163d98146cb190eb2952e4e250b30ed2b8b501ba" exitCode=0
Oct 30 22:08:44.713213 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.713212    4477 generic.go:356] "Generic (PLEG): container finished" podID="f904edd0-10dc-4958-8d9b-9e012c1319db" containerID="6cd67108b4e7516c87b9a3cadbfd54edd7d436841f49ce876fc03d7d718ca2ec" exitCode=0
Oct 30 22:08:44.713213 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.713218    4477 generic.go:356] "Generic (PLEG): container finished" podID="f904edd0-10dc-4958-8d9b-9e012c1319db" containerID="1b3367b9f00d774fd20bfb6349716fbc15f118a8d18bbfc4d0fd9af8c72c63f7" exitCode=0
Oct 30 22:08:44.713213 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.713223    4477 generic.go:356] "Generic (PLEG): container finished" podID="f904edd0-10dc-4958-8d9b-9e012c1319db" containerID="c0c298ca2a98a892f23adecaddc83575f89fb6ed37cd952300aa86954ca586fb" exitCode=0
Oct 30 22:08:44.713602 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.713228    4477 generic.go:356] "Generic (PLEG): container finished" podID="f904edd0-10dc-4958-8d9b-9e012c1319db" containerID="e3cd27a71f050df532a08814a4280c80556187adb7c8d22fa82cd343369a30e0" exitCode=0
Oct 30 22:08:44.713602 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.713259    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f904edd0-10dc-4958-8d9b-9e012c1319db","Type":"ContainerDied","Data":"e59f8a472d1f4bf287761c5eccf03ceed9f5b1b9243e84a9dd2cf9faa9c9801a"}
Oct 30 22:08:44.713602 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.713280    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:08:44.713602 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.713330    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f904edd0-10dc-4958-8d9b-9e012c1319db","Type":"ContainerDied","Data":"0fe9c11ae206823bc898cf13163d98146cb190eb2952e4e250b30ed2b8b501ba"}
Oct 30 22:08:44.713602 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.713357    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f904edd0-10dc-4958-8d9b-9e012c1319db","Type":"ContainerDied","Data":"6cd67108b4e7516c87b9a3cadbfd54edd7d436841f49ce876fc03d7d718ca2ec"}
Oct 30 22:08:44.713602 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.713382    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f904edd0-10dc-4958-8d9b-9e012c1319db","Type":"ContainerDied","Data":"1b3367b9f00d774fd20bfb6349716fbc15f118a8d18bbfc4d0fd9af8c72c63f7"}
Oct 30 22:08:44.713602 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.713397    4477 scope.go:122] "RemoveContainer" containerID="e59f8a472d1f4bf287761c5eccf03ceed9f5b1b9243e84a9dd2cf9faa9c9801a"
Oct 30 22:08:44.713602 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.713407    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f904edd0-10dc-4958-8d9b-9e012c1319db","Type":"ContainerDied","Data":"c0c298ca2a98a892f23adecaddc83575f89fb6ed37cd952300aa86954ca586fb"}
Oct 30 22:08:44.713602 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.713431    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f904edd0-10dc-4958-8d9b-9e012c1319db","Type":"ContainerDied","Data":"e3cd27a71f050df532a08814a4280c80556187adb7c8d22fa82cd343369a30e0"}
Oct 30 22:08:44.713602 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.713454    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f904edd0-10dc-4958-8d9b-9e012c1319db","Type":"ContainerDied","Data":"1a5bfee254bde827ac9f688534def5ec0361ccac1bf79eb00e45d3e5c073e5e9"}
Oct 30 22:08:44.721120 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.721100    4477 scope.go:122] "RemoveContainer" containerID="0fe9c11ae206823bc898cf13163d98146cb190eb2952e4e250b30ed2b8b501ba"
Oct 30 22:08:44.727766 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.727751    4477 scope.go:122] "RemoveContainer" containerID="6cd67108b4e7516c87b9a3cadbfd54edd7d436841f49ce876fc03d7d718ca2ec"
Oct 30 22:08:44.734200 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.734186    4477 scope.go:122] "RemoveContainer" containerID="1b3367b9f00d774fd20bfb6349716fbc15f118a8d18bbfc4d0fd9af8c72c63f7"
Oct 30 22:08:44.741028 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.741012    4477 scope.go:122] "RemoveContainer" containerID="c0c298ca2a98a892f23adecaddc83575f89fb6ed37cd952300aa86954ca586fb"
Oct 30 22:08:44.747371 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.747356    4477 scope.go:122] "RemoveContainer" containerID="e3cd27a71f050df532a08814a4280c80556187adb7c8d22fa82cd343369a30e0"
Oct 30 22:08:44.754624 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.754609    4477 scope.go:122] "RemoveContainer" containerID="ab20b6db94f1e4cda42e0b032c7c382dea247c3c780f4399c70a7eb339cd66a6"
Oct 30 22:08:44.760774 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.760759    4477 scope.go:122] "RemoveContainer" containerID="e59f8a472d1f4bf287761c5eccf03ceed9f5b1b9243e84a9dd2cf9faa9c9801a"
Oct 30 22:08:44.761016 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:08:44.760999    4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e59f8a472d1f4bf287761c5eccf03ceed9f5b1b9243e84a9dd2cf9faa9c9801a\": container with ID starting with e59f8a472d1f4bf287761c5eccf03ceed9f5b1b9243e84a9dd2cf9faa9c9801a not found: ID does not exist" containerID="e59f8a472d1f4bf287761c5eccf03ceed9f5b1b9243e84a9dd2cf9faa9c9801a"
Oct 30 22:08:44.761092 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.761024    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e59f8a472d1f4bf287761c5eccf03ceed9f5b1b9243e84a9dd2cf9faa9c9801a"} err="failed to get container status \"e59f8a472d1f4bf287761c5eccf03ceed9f5b1b9243e84a9dd2cf9faa9c9801a\": rpc error: code = NotFound desc = could not find container \"e59f8a472d1f4bf287761c5eccf03ceed9f5b1b9243e84a9dd2cf9faa9c9801a\": container with ID starting with e59f8a472d1f4bf287761c5eccf03ceed9f5b1b9243e84a9dd2cf9faa9c9801a not found: ID does not exist"
Oct 30 22:08:44.761092 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.761054    4477 scope.go:122] "RemoveContainer" containerID="0fe9c11ae206823bc898cf13163d98146cb190eb2952e4e250b30ed2b8b501ba"
Oct 30 22:08:44.761317 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:08:44.761301    4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fe9c11ae206823bc898cf13163d98146cb190eb2952e4e250b30ed2b8b501ba\": container with ID starting with 0fe9c11ae206823bc898cf13163d98146cb190eb2952e4e250b30ed2b8b501ba not found: ID does not exist" containerID="0fe9c11ae206823bc898cf13163d98146cb190eb2952e4e250b30ed2b8b501ba"
Oct 30 22:08:44.761354 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.761323    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fe9c11ae206823bc898cf13163d98146cb190eb2952e4e250b30ed2b8b501ba"} err="failed to get container status \"0fe9c11ae206823bc898cf13163d98146cb190eb2952e4e250b30ed2b8b501ba\": rpc error: code = NotFound desc = could not find container \"0fe9c11ae206823bc898cf13163d98146cb190eb2952e4e250b30ed2b8b501ba\": container with ID starting with 0fe9c11ae206823bc898cf13163d98146cb190eb2952e4e250b30ed2b8b501ba not found: ID does not exist"
Oct 30 22:08:44.761354 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.761337    4477 scope.go:122] "RemoveContainer" containerID="6cd67108b4e7516c87b9a3cadbfd54edd7d436841f49ce876fc03d7d718ca2ec"
Oct 30 22:08:44.761587 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:08:44.761543    4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cd67108b4e7516c87b9a3cadbfd54edd7d436841f49ce876fc03d7d718ca2ec\": container with ID starting with 6cd67108b4e7516c87b9a3cadbfd54edd7d436841f49ce876fc03d7d718ca2ec not found: ID does not exist" containerID="6cd67108b4e7516c87b9a3cadbfd54edd7d436841f49ce876fc03d7d718ca2ec"
Oct 30 22:08:44.761632 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.761592    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cd67108b4e7516c87b9a3cadbfd54edd7d436841f49ce876fc03d7d718ca2ec"} err="failed to get container status \"6cd67108b4e7516c87b9a3cadbfd54edd7d436841f49ce876fc03d7d718ca2ec\": rpc error: code = NotFound desc = could not find container \"6cd67108b4e7516c87b9a3cadbfd54edd7d436841f49ce876fc03d7d718ca2ec\": container with ID starting with 6cd67108b4e7516c87b9a3cadbfd54edd7d436841f49ce876fc03d7d718ca2ec not found: ID does not exist"
Oct 30 22:08:44.761632 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.761606    4477 scope.go:122] "RemoveContainer" containerID="1b3367b9f00d774fd20bfb6349716fbc15f118a8d18bbfc4d0fd9af8c72c63f7"
Oct 30 22:08:44.761827 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:08:44.761811    4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b3367b9f00d774fd20bfb6349716fbc15f118a8d18bbfc4d0fd9af8c72c63f7\": container with ID starting with 1b3367b9f00d774fd20bfb6349716fbc15f118a8d18bbfc4d0fd9af8c72c63f7 not found: ID does not exist" containerID="1b3367b9f00d774fd20bfb6349716fbc15f118a8d18bbfc4d0fd9af8c72c63f7"
Oct 30 22:08:44.761858 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.761833    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b3367b9f00d774fd20bfb6349716fbc15f118a8d18bbfc4d0fd9af8c72c63f7"} err="failed to get container status \"1b3367b9f00d774fd20bfb6349716fbc15f118a8d18bbfc4d0fd9af8c72c63f7\": rpc error: code = NotFound desc = could not find container \"1b3367b9f00d774fd20bfb6349716fbc15f118a8d18bbfc4d0fd9af8c72c63f7\": container with ID starting with 1b3367b9f00d774fd20bfb6349716fbc15f118a8d18bbfc4d0fd9af8c72c63f7 not found: ID does not exist"
Oct 30 22:08:44.761858 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.761847    4477 scope.go:122] "RemoveContainer" containerID="c0c298ca2a98a892f23adecaddc83575f89fb6ed37cd952300aa86954ca586fb"
Oct 30 22:08:44.762076 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:08:44.762059    4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0c298ca2a98a892f23adecaddc83575f89fb6ed37cd952300aa86954ca586fb\": container with ID starting with c0c298ca2a98a892f23adecaddc83575f89fb6ed37cd952300aa86954ca586fb not found: ID does not exist" containerID="c0c298ca2a98a892f23adecaddc83575f89fb6ed37cd952300aa86954ca586fb"
Oct 30 22:08:44.762136 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.762083    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0c298ca2a98a892f23adecaddc83575f89fb6ed37cd952300aa86954ca586fb"} err="failed to get container status \"c0c298ca2a98a892f23adecaddc83575f89fb6ed37cd952300aa86954ca586fb\": rpc error: code = NotFound desc = could not find container \"c0c298ca2a98a892f23adecaddc83575f89fb6ed37cd952300aa86954ca586fb\": container with ID starting with c0c298ca2a98a892f23adecaddc83575f89fb6ed37cd952300aa86954ca586fb not found: ID does not exist"
Oct 30 22:08:44.762136 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.762102    4477 scope.go:122] "RemoveContainer" containerID="e3cd27a71f050df532a08814a4280c80556187adb7c8d22fa82cd343369a30e0"
Oct 30 22:08:44.762326 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:08:44.762311    4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3cd27a71f050df532a08814a4280c80556187adb7c8d22fa82cd343369a30e0\": container with ID starting with e3cd27a71f050df532a08814a4280c80556187adb7c8d22fa82cd343369a30e0 not found: ID does not exist" containerID="e3cd27a71f050df532a08814a4280c80556187adb7c8d22fa82cd343369a30e0"
Oct 30 22:08:44.762386 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.762332    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3cd27a71f050df532a08814a4280c80556187adb7c8d22fa82cd343369a30e0"} err="failed to get container status \"e3cd27a71f050df532a08814a4280c80556187adb7c8d22fa82cd343369a30e0\": rpc error: code = NotFound desc = could not find container \"e3cd27a71f050df532a08814a4280c80556187adb7c8d22fa82cd343369a30e0\": container with ID starting with e3cd27a71f050df532a08814a4280c80556187adb7c8d22fa82cd343369a30e0 not found: ID does not exist"
Oct 30 22:08:44.762386 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.762351    4477 scope.go:122] "RemoveContainer" containerID="ab20b6db94f1e4cda42e0b032c7c382dea247c3c780f4399c70a7eb339cd66a6"
Oct 30 22:08:44.762576 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:08:44.762558    4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab20b6db94f1e4cda42e0b032c7c382dea247c3c780f4399c70a7eb339cd66a6\": container with ID starting with ab20b6db94f1e4cda42e0b032c7c382dea247c3c780f4399c70a7eb339cd66a6 not found: ID does not exist" containerID="ab20b6db94f1e4cda42e0b032c7c382dea247c3c780f4399c70a7eb339cd66a6"
Oct 30 22:08:44.762609 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.762581    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab20b6db94f1e4cda42e0b032c7c382dea247c3c780f4399c70a7eb339cd66a6"} err="failed to get container status \"ab20b6db94f1e4cda42e0b032c7c382dea247c3c780f4399c70a7eb339cd66a6\": rpc error: code = NotFound desc = could not find container \"ab20b6db94f1e4cda42e0b032c7c382dea247c3c780f4399c70a7eb339cd66a6\": container with ID starting with ab20b6db94f1e4cda42e0b032c7c382dea247c3c780f4399c70a7eb339cd66a6 not found: ID does not exist"
Oct 30 22:08:44.762609 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.762595    4477 scope.go:122] "RemoveContainer" containerID="e59f8a472d1f4bf287761c5eccf03ceed9f5b1b9243e84a9dd2cf9faa9c9801a"
Oct 30 22:08:44.762819 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.762800    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e59f8a472d1f4bf287761c5eccf03ceed9f5b1b9243e84a9dd2cf9faa9c9801a"} err="failed to get container status \"e59f8a472d1f4bf287761c5eccf03ceed9f5b1b9243e84a9dd2cf9faa9c9801a\": rpc error: code = NotFound desc = could not find container \"e59f8a472d1f4bf287761c5eccf03ceed9f5b1b9243e84a9dd2cf9faa9c9801a\": container with ID starting with e59f8a472d1f4bf287761c5eccf03ceed9f5b1b9243e84a9dd2cf9faa9c9801a not found: ID does not exist"
Oct 30 22:08:44.762888 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.762819    4477 scope.go:122] "RemoveContainer" containerID="0fe9c11ae206823bc898cf13163d98146cb190eb2952e4e250b30ed2b8b501ba"
Oct 30 22:08:44.763036 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.763019    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fe9c11ae206823bc898cf13163d98146cb190eb2952e4e250b30ed2b8b501ba"} err="failed to get container status \"0fe9c11ae206823bc898cf13163d98146cb190eb2952e4e250b30ed2b8b501ba\": rpc error: code = NotFound desc = could not find container \"0fe9c11ae206823bc898cf13163d98146cb190eb2952e4e250b30ed2b8b501ba\": container with ID starting with 0fe9c11ae206823bc898cf13163d98146cb190eb2952e4e250b30ed2b8b501ba not found: ID does not exist"
Oct 30 22:08:44.763134 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.763123    4477 scope.go:122] "RemoveContainer" containerID="6cd67108b4e7516c87b9a3cadbfd54edd7d436841f49ce876fc03d7d718ca2ec"
Oct 30 22:08:44.763347 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.763328    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cd67108b4e7516c87b9a3cadbfd54edd7d436841f49ce876fc03d7d718ca2ec"} err="failed to get container status \"6cd67108b4e7516c87b9a3cadbfd54edd7d436841f49ce876fc03d7d718ca2ec\": rpc error: code = NotFound desc = could not find container \"6cd67108b4e7516c87b9a3cadbfd54edd7d436841f49ce876fc03d7d718ca2ec\": container with ID starting with 6cd67108b4e7516c87b9a3cadbfd54edd7d436841f49ce876fc03d7d718ca2ec not found: ID does not exist"
Oct 30 22:08:44.763420 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.763348    4477 scope.go:122] "RemoveContainer" containerID="1b3367b9f00d774fd20bfb6349716fbc15f118a8d18bbfc4d0fd9af8c72c63f7"
Oct 30 22:08:44.763557 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.763534    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b3367b9f00d774fd20bfb6349716fbc15f118a8d18bbfc4d0fd9af8c72c63f7"} err="failed to get container status \"1b3367b9f00d774fd20bfb6349716fbc15f118a8d18bbfc4d0fd9af8c72c63f7\": rpc error: code = NotFound desc = could not find container \"1b3367b9f00d774fd20bfb6349716fbc15f118a8d18bbfc4d0fd9af8c72c63f7\": container with ID starting with 1b3367b9f00d774fd20bfb6349716fbc15f118a8d18bbfc4d0fd9af8c72c63f7 not found: ID does not exist"
Oct 30 22:08:44.763602 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.763558    4477 scope.go:122] "RemoveContainer" containerID="c0c298ca2a98a892f23adecaddc83575f89fb6ed37cd952300aa86954ca586fb"
Oct 30 22:08:44.763727 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.763709    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0c298ca2a98a892f23adecaddc83575f89fb6ed37cd952300aa86954ca586fb"} err="failed to get container status \"c0c298ca2a98a892f23adecaddc83575f89fb6ed37cd952300aa86954ca586fb\": rpc error: code = NotFound desc = could not find container \"c0c298ca2a98a892f23adecaddc83575f89fb6ed37cd952300aa86954ca586fb\": container with ID starting with c0c298ca2a98a892f23adecaddc83575f89fb6ed37cd952300aa86954ca586fb not found: ID does not exist"
Oct 30 22:08:44.763793 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.763727    4477 scope.go:122] "RemoveContainer" containerID="e3cd27a71f050df532a08814a4280c80556187adb7c8d22fa82cd343369a30e0"
Oct 30 22:08:44.763926 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.763911    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3cd27a71f050df532a08814a4280c80556187adb7c8d22fa82cd343369a30e0"} err="failed to get container status \"e3cd27a71f050df532a08814a4280c80556187adb7c8d22fa82cd343369a30e0\": rpc error: code = NotFound desc = could not find container \"e3cd27a71f050df532a08814a4280c80556187adb7c8d22fa82cd343369a30e0\": container with ID starting with e3cd27a71f050df532a08814a4280c80556187adb7c8d22fa82cd343369a30e0 not found: ID does not exist"
Oct 30 22:08:44.763970 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.763927    4477 scope.go:122] "RemoveContainer" containerID="ab20b6db94f1e4cda42e0b032c7c382dea247c3c780f4399c70a7eb339cd66a6"
Oct 30 22:08:44.764188 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.764169    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab20b6db94f1e4cda42e0b032c7c382dea247c3c780f4399c70a7eb339cd66a6"} err="failed to get container status \"ab20b6db94f1e4cda42e0b032c7c382dea247c3c780f4399c70a7eb339cd66a6\": rpc error: code = NotFound desc = could not find container \"ab20b6db94f1e4cda42e0b032c7c382dea247c3c780f4399c70a7eb339cd66a6\": container with ID starting with ab20b6db94f1e4cda42e0b032c7c382dea247c3c780f4399c70a7eb339cd66a6 not found: ID does not exist"
Oct 30 22:08:44.764268 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.764187    4477 scope.go:122] "RemoveContainer" containerID="e59f8a472d1f4bf287761c5eccf03ceed9f5b1b9243e84a9dd2cf9faa9c9801a"
Oct 30 22:08:44.764423 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.764405    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e59f8a472d1f4bf287761c5eccf03ceed9f5b1b9243e84a9dd2cf9faa9c9801a"} err="failed to get container status \"e59f8a472d1f4bf287761c5eccf03ceed9f5b1b9243e84a9dd2cf9faa9c9801a\": rpc error: code = NotFound desc = could not find container \"e59f8a472d1f4bf287761c5eccf03ceed9f5b1b9243e84a9dd2cf9faa9c9801a\": container with ID starting with e59f8a472d1f4bf287761c5eccf03ceed9f5b1b9243e84a9dd2cf9faa9c9801a not found: ID does not exist"
Oct 30 22:08:44.764465 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.764424    4477 scope.go:122] "RemoveContainer" containerID="0fe9c11ae206823bc898cf13163d98146cb190eb2952e4e250b30ed2b8b501ba"
Oct 30 22:08:44.764627 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.764609    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fe9c11ae206823bc898cf13163d98146cb190eb2952e4e250b30ed2b8b501ba"} err="failed to get container status \"0fe9c11ae206823bc898cf13163d98146cb190eb2952e4e250b30ed2b8b501ba\": rpc error: code = NotFound desc = could not find container \"0fe9c11ae206823bc898cf13163d98146cb190eb2952e4e250b30ed2b8b501ba\": container with ID starting with 0fe9c11ae206823bc898cf13163d98146cb190eb2952e4e250b30ed2b8b501ba not found: ID does not exist"
Oct 30 22:08:44.764701 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.764628    4477 scope.go:122] "RemoveContainer" containerID="6cd67108b4e7516c87b9a3cadbfd54edd7d436841f49ce876fc03d7d718ca2ec"
Oct 30 22:08:44.764833 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.764817    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cd67108b4e7516c87b9a3cadbfd54edd7d436841f49ce876fc03d7d718ca2ec"} err="failed to get container status \"6cd67108b4e7516c87b9a3cadbfd54edd7d436841f49ce876fc03d7d718ca2ec\": rpc error: code = NotFound desc = could not find container \"6cd67108b4e7516c87b9a3cadbfd54edd7d436841f49ce876fc03d7d718ca2ec\": container with ID starting with 6cd67108b4e7516c87b9a3cadbfd54edd7d436841f49ce876fc03d7d718ca2ec not found: ID does not exist"
Oct 30 22:08:44.764878 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.764833    4477 scope.go:122] "RemoveContainer" containerID="1b3367b9f00d774fd20bfb6349716fbc15f118a8d18bbfc4d0fd9af8c72c63f7"
Oct 30 22:08:44.765025 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.765007    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b3367b9f00d774fd20bfb6349716fbc15f118a8d18bbfc4d0fd9af8c72c63f7"} err="failed to get container status \"1b3367b9f00d774fd20bfb6349716fbc15f118a8d18bbfc4d0fd9af8c72c63f7\": rpc error: code = NotFound desc = could not find container \"1b3367b9f00d774fd20bfb6349716fbc15f118a8d18bbfc4d0fd9af8c72c63f7\": container with ID starting with 1b3367b9f00d774fd20bfb6349716fbc15f118a8d18bbfc4d0fd9af8c72c63f7 not found: ID does not exist"
Oct 30 22:08:44.765099 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.765025    4477 scope.go:122] "RemoveContainer" containerID="c0c298ca2a98a892f23adecaddc83575f89fb6ed37cd952300aa86954ca586fb"
Oct 30 22:08:44.765243 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.765226    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0c298ca2a98a892f23adecaddc83575f89fb6ed37cd952300aa86954ca586fb"} err="failed to get container status \"c0c298ca2a98a892f23adecaddc83575f89fb6ed37cd952300aa86954ca586fb\": rpc error: code = NotFound desc = could not find container \"c0c298ca2a98a892f23adecaddc83575f89fb6ed37cd952300aa86954ca586fb\": container with ID starting with c0c298ca2a98a892f23adecaddc83575f89fb6ed37cd952300aa86954ca586fb not found: ID does not exist"
Oct 30 22:08:44.765284 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.765243    4477 scope.go:122] "RemoveContainer" containerID="e3cd27a71f050df532a08814a4280c80556187adb7c8d22fa82cd343369a30e0"
Oct 30 22:08:44.765449 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.765432    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3cd27a71f050df532a08814a4280c80556187adb7c8d22fa82cd343369a30e0"} err="failed to get container status \"e3cd27a71f050df532a08814a4280c80556187adb7c8d22fa82cd343369a30e0\": rpc error: code = NotFound desc = could not find container \"e3cd27a71f050df532a08814a4280c80556187adb7c8d22fa82cd343369a30e0\": container with ID starting with e3cd27a71f050df532a08814a4280c80556187adb7c8d22fa82cd343369a30e0 not found: ID does not exist"
Oct 30 22:08:44.765509 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.765449    4477 scope.go:122] "RemoveContainer" containerID="ab20b6db94f1e4cda42e0b032c7c382dea247c3c780f4399c70a7eb339cd66a6"
Oct 30 22:08:44.765667 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.765649    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab20b6db94f1e4cda42e0b032c7c382dea247c3c780f4399c70a7eb339cd66a6"} err="failed to get container status \"ab20b6db94f1e4cda42e0b032c7c382dea247c3c780f4399c70a7eb339cd66a6\": rpc error: code = NotFound desc = could not find container \"ab20b6db94f1e4cda42e0b032c7c382dea247c3c780f4399c70a7eb339cd66a6\": container with ID starting with ab20b6db94f1e4cda42e0b032c7c382dea247c3c780f4399c70a7eb339cd66a6 not found: ID does not exist"
Oct 30 22:08:44.765707 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.765667    4477 scope.go:122] "RemoveContainer" containerID="e59f8a472d1f4bf287761c5eccf03ceed9f5b1b9243e84a9dd2cf9faa9c9801a"
Oct 30 22:08:44.765884 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.765866    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e59f8a472d1f4bf287761c5eccf03ceed9f5b1b9243e84a9dd2cf9faa9c9801a"} err="failed to get container status \"e59f8a472d1f4bf287761c5eccf03ceed9f5b1b9243e84a9dd2cf9faa9c9801a\": rpc error: code = NotFound desc = could not find container \"e59f8a472d1f4bf287761c5eccf03ceed9f5b1b9243e84a9dd2cf9faa9c9801a\": container with ID starting with e59f8a472d1f4bf287761c5eccf03ceed9f5b1b9243e84a9dd2cf9faa9c9801a not found: ID does not exist"
Oct 30 22:08:44.765925 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.765884    4477 scope.go:122] "RemoveContainer" containerID="0fe9c11ae206823bc898cf13163d98146cb190eb2952e4e250b30ed2b8b501ba"
Oct 30 22:08:44.766109 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.766092    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fe9c11ae206823bc898cf13163d98146cb190eb2952e4e250b30ed2b8b501ba"} err="failed to get container status \"0fe9c11ae206823bc898cf13163d98146cb190eb2952e4e250b30ed2b8b501ba\": rpc error: code = NotFound desc = could not find container \"0fe9c11ae206823bc898cf13163d98146cb190eb2952e4e250b30ed2b8b501ba\": container with ID starting with 0fe9c11ae206823bc898cf13163d98146cb190eb2952e4e250b30ed2b8b501ba not found: ID does not exist"
Oct 30 22:08:44.766159 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.766109    4477 scope.go:122] "RemoveContainer" containerID="6cd67108b4e7516c87b9a3cadbfd54edd7d436841f49ce876fc03d7d718ca2ec"
Oct 30 22:08:44.766336 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.766318    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cd67108b4e7516c87b9a3cadbfd54edd7d436841f49ce876fc03d7d718ca2ec"} err="failed to get container status \"6cd67108b4e7516c87b9a3cadbfd54edd7d436841f49ce876fc03d7d718ca2ec\": rpc error: code = NotFound desc = could not find container \"6cd67108b4e7516c87b9a3cadbfd54edd7d436841f49ce876fc03d7d718ca2ec\": container with ID starting with 6cd67108b4e7516c87b9a3cadbfd54edd7d436841f49ce876fc03d7d718ca2ec not found: ID does not exist"
Oct 30 22:08:44.766378 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.766336    4477 scope.go:122] "RemoveContainer" containerID="1b3367b9f00d774fd20bfb6349716fbc15f118a8d18bbfc4d0fd9af8c72c63f7"
Oct 30 22:08:44.766540 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.766519    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b3367b9f00d774fd20bfb6349716fbc15f118a8d18bbfc4d0fd9af8c72c63f7"} err="failed to get container status \"1b3367b9f00d774fd20bfb6349716fbc15f118a8d18bbfc4d0fd9af8c72c63f7\": rpc error: code = NotFound desc = could not find container \"1b3367b9f00d774fd20bfb6349716fbc15f118a8d18bbfc4d0fd9af8c72c63f7\": container with ID starting with 1b3367b9f00d774fd20bfb6349716fbc15f118a8d18bbfc4d0fd9af8c72c63f7 not found: ID does not exist"
Oct 30 22:08:44.766608 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.766541    4477 scope.go:122] "RemoveContainer" containerID="c0c298ca2a98a892f23adecaddc83575f89fb6ed37cd952300aa86954ca586fb"
Oct 30 22:08:44.766766 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.766747    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0c298ca2a98a892f23adecaddc83575f89fb6ed37cd952300aa86954ca586fb"} err="failed to get container status \"c0c298ca2a98a892f23adecaddc83575f89fb6ed37cd952300aa86954ca586fb\": rpc error: code = NotFound desc = could not find container \"c0c298ca2a98a892f23adecaddc83575f89fb6ed37cd952300aa86954ca586fb\": container with ID starting with c0c298ca2a98a892f23adecaddc83575f89fb6ed37cd952300aa86954ca586fb not found: ID does not exist"
Oct 30 22:08:44.766829 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.766767    4477 scope.go:122] "RemoveContainer" containerID="e3cd27a71f050df532a08814a4280c80556187adb7c8d22fa82cd343369a30e0"
Oct 30 22:08:44.766989 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.766972    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3cd27a71f050df532a08814a4280c80556187adb7c8d22fa82cd343369a30e0"} err="failed to get container status \"e3cd27a71f050df532a08814a4280c80556187adb7c8d22fa82cd343369a30e0\": rpc error: code = NotFound desc = could not find container \"e3cd27a71f050df532a08814a4280c80556187adb7c8d22fa82cd343369a30e0\": container with ID starting with e3cd27a71f050df532a08814a4280c80556187adb7c8d22fa82cd343369a30e0 not found: ID does not exist"
Oct 30 22:08:44.767029 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.766989    4477 scope.go:122] "RemoveContainer" containerID="ab20b6db94f1e4cda42e0b032c7c382dea247c3c780f4399c70a7eb339cd66a6"
Oct 30 22:08:44.767211 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.767193    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab20b6db94f1e4cda42e0b032c7c382dea247c3c780f4399c70a7eb339cd66a6"} err="failed to get container status \"ab20b6db94f1e4cda42e0b032c7c382dea247c3c780f4399c70a7eb339cd66a6\": rpc error: code = NotFound desc = could not find container \"ab20b6db94f1e4cda42e0b032c7c382dea247c3c780f4399c70a7eb339cd66a6\": container with ID starting with ab20b6db94f1e4cda42e0b032c7c382dea247c3c780f4399c70a7eb339cd66a6 not found: ID does not exist"
Oct 30 22:08:44.767281 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.767212    4477 scope.go:122] "RemoveContainer" containerID="e59f8a472d1f4bf287761c5eccf03ceed9f5b1b9243e84a9dd2cf9faa9c9801a"
Oct 30 22:08:44.767418 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.767399    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e59f8a472d1f4bf287761c5eccf03ceed9f5b1b9243e84a9dd2cf9faa9c9801a"} err="failed to get container status \"e59f8a472d1f4bf287761c5eccf03ceed9f5b1b9243e84a9dd2cf9faa9c9801a\": rpc error: code = NotFound desc = could not find container \"e59f8a472d1f4bf287761c5eccf03ceed9f5b1b9243e84a9dd2cf9faa9c9801a\": container with ID starting with e59f8a472d1f4bf287761c5eccf03ceed9f5b1b9243e84a9dd2cf9faa9c9801a not found: ID does not exist"
Oct 30 22:08:44.767459 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.767420    4477 scope.go:122] "RemoveContainer" containerID="0fe9c11ae206823bc898cf13163d98146cb190eb2952e4e250b30ed2b8b501ba"
Oct 30 22:08:44.767603 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.767584    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fe9c11ae206823bc898cf13163d98146cb190eb2952e4e250b30ed2b8b501ba"} err="failed to get container status \"0fe9c11ae206823bc898cf13163d98146cb190eb2952e4e250b30ed2b8b501ba\": rpc error: code = NotFound desc = could not find container \"0fe9c11ae206823bc898cf13163d98146cb190eb2952e4e250b30ed2b8b501ba\": container with ID starting with 0fe9c11ae206823bc898cf13163d98146cb190eb2952e4e250b30ed2b8b501ba not found: ID does not exist"
Oct 30 22:08:44.767639 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.767604    4477 scope.go:122] "RemoveContainer" containerID="6cd67108b4e7516c87b9a3cadbfd54edd7d436841f49ce876fc03d7d718ca2ec"
Oct 30 22:08:44.767775 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.767760    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cd67108b4e7516c87b9a3cadbfd54edd7d436841f49ce876fc03d7d718ca2ec"} err="failed to get container status \"6cd67108b4e7516c87b9a3cadbfd54edd7d436841f49ce876fc03d7d718ca2ec\": rpc error: code = NotFound desc = could not find container \"6cd67108b4e7516c87b9a3cadbfd54edd7d436841f49ce876fc03d7d718ca2ec\": container with ID starting with 6cd67108b4e7516c87b9a3cadbfd54edd7d436841f49ce876fc03d7d718ca2ec not found: ID does not exist"
Oct 30 22:08:44.767817 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.767775    4477 scope.go:122] "RemoveContainer" containerID="1b3367b9f00d774fd20bfb6349716fbc15f118a8d18bbfc4d0fd9af8c72c63f7"
Oct 30 22:08:44.768006 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.767989    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b3367b9f00d774fd20bfb6349716fbc15f118a8d18bbfc4d0fd9af8c72c63f7"} err="failed to get container status \"1b3367b9f00d774fd20bfb6349716fbc15f118a8d18bbfc4d0fd9af8c72c63f7\": rpc error: code = NotFound desc = could not find container \"1b3367b9f00d774fd20bfb6349716fbc15f118a8d18bbfc4d0fd9af8c72c63f7\": container with ID starting with 1b3367b9f00d774fd20bfb6349716fbc15f118a8d18bbfc4d0fd9af8c72c63f7 not found: ID does not exist"
Oct 30 22:08:44.768065 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.768007    4477 scope.go:122] "RemoveContainer" containerID="c0c298ca2a98a892f23adecaddc83575f89fb6ed37cd952300aa86954ca586fb"
Oct 30 22:08:44.768265 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.768245    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0c298ca2a98a892f23adecaddc83575f89fb6ed37cd952300aa86954ca586fb"} err="failed to get container status \"c0c298ca2a98a892f23adecaddc83575f89fb6ed37cd952300aa86954ca586fb\": rpc error: code = NotFound desc = could not find container \"c0c298ca2a98a892f23adecaddc83575f89fb6ed37cd952300aa86954ca586fb\": container with ID starting with c0c298ca2a98a892f23adecaddc83575f89fb6ed37cd952300aa86954ca586fb not found: ID does not exist"
Oct 30 22:08:44.768314 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.768266    4477 scope.go:122] "RemoveContainer" containerID="e3cd27a71f050df532a08814a4280c80556187adb7c8d22fa82cd343369a30e0"
Oct 30 22:08:44.768454 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.768439    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3cd27a71f050df532a08814a4280c80556187adb7c8d22fa82cd343369a30e0"} err="failed to get container status \"e3cd27a71f050df532a08814a4280c80556187adb7c8d22fa82cd343369a30e0\": rpc error: code = NotFound desc = could not find container \"e3cd27a71f050df532a08814a4280c80556187adb7c8d22fa82cd343369a30e0\": container with ID starting with e3cd27a71f050df532a08814a4280c80556187adb7c8d22fa82cd343369a30e0 not found: ID does not exist"
Oct 30 22:08:44.768454 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.768453    4477 scope.go:122] "RemoveContainer" containerID="ab20b6db94f1e4cda42e0b032c7c382dea247c3c780f4399c70a7eb339cd66a6"
Oct 30 22:08:44.768678 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.768653    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab20b6db94f1e4cda42e0b032c7c382dea247c3c780f4399c70a7eb339cd66a6"} err="failed to get container status \"ab20b6db94f1e4cda42e0b032c7c382dea247c3c780f4399c70a7eb339cd66a6\": rpc error: code = NotFound desc = could not find container \"ab20b6db94f1e4cda42e0b032c7c382dea247c3c780f4399c70a7eb339cd66a6\": container with ID starting with ab20b6db94f1e4cda42e0b032c7c382dea247c3c780f4399c70a7eb339cd66a6 not found: ID does not exist"
Oct 30 22:08:44.768745 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.768680    4477 scope.go:122] "RemoveContainer" containerID="e59f8a472d1f4bf287761c5eccf03ceed9f5b1b9243e84a9dd2cf9faa9c9801a"
Oct 30 22:08:44.768893 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.768879    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e59f8a472d1f4bf287761c5eccf03ceed9f5b1b9243e84a9dd2cf9faa9c9801a"} err="failed to get container status \"e59f8a472d1f4bf287761c5eccf03ceed9f5b1b9243e84a9dd2cf9faa9c9801a\": rpc error: code = NotFound desc = could not find container \"e59f8a472d1f4bf287761c5eccf03ceed9f5b1b9243e84a9dd2cf9faa9c9801a\": container with ID starting with e59f8a472d1f4bf287761c5eccf03ceed9f5b1b9243e84a9dd2cf9faa9c9801a not found: ID does not exist"
Oct 30 22:08:44.768938 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.768893    4477 scope.go:122] "RemoveContainer" containerID="0fe9c11ae206823bc898cf13163d98146cb190eb2952e4e250b30ed2b8b501ba"
Oct 30 22:08:44.769124 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.769105    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fe9c11ae206823bc898cf13163d98146cb190eb2952e4e250b30ed2b8b501ba"} err="failed to get container status \"0fe9c11ae206823bc898cf13163d98146cb190eb2952e4e250b30ed2b8b501ba\": rpc error: code = NotFound desc = could not find container \"0fe9c11ae206823bc898cf13163d98146cb190eb2952e4e250b30ed2b8b501ba\": container with ID starting with 0fe9c11ae206823bc898cf13163d98146cb190eb2952e4e250b30ed2b8b501ba not found: ID does not exist"
Oct 30 22:08:44.769189 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.769125    4477 scope.go:122] "RemoveContainer" containerID="6cd67108b4e7516c87b9a3cadbfd54edd7d436841f49ce876fc03d7d718ca2ec"
Oct 30 22:08:44.769361 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.769343    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cd67108b4e7516c87b9a3cadbfd54edd7d436841f49ce876fc03d7d718ca2ec"} err="failed to get container status \"6cd67108b4e7516c87b9a3cadbfd54edd7d436841f49ce876fc03d7d718ca2ec\": rpc error: code = NotFound desc = could not find container \"6cd67108b4e7516c87b9a3cadbfd54edd7d436841f49ce876fc03d7d718ca2ec\": container with ID starting with 6cd67108b4e7516c87b9a3cadbfd54edd7d436841f49ce876fc03d7d718ca2ec not found: ID does not exist"
Oct 30 22:08:44.769408 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.769361    4477 scope.go:122] "RemoveContainer" containerID="1b3367b9f00d774fd20bfb6349716fbc15f118a8d18bbfc4d0fd9af8c72c63f7"
Oct 30 22:08:44.769575 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.769548    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b3367b9f00d774fd20bfb6349716fbc15f118a8d18bbfc4d0fd9af8c72c63f7"} err="failed to get container status \"1b3367b9f00d774fd20bfb6349716fbc15f118a8d18bbfc4d0fd9af8c72c63f7\": rpc error: code = NotFound desc = could not find container \"1b3367b9f00d774fd20bfb6349716fbc15f118a8d18bbfc4d0fd9af8c72c63f7\": container with ID starting with 1b3367b9f00d774fd20bfb6349716fbc15f118a8d18bbfc4d0fd9af8c72c63f7 not found: ID does not exist"
Oct 30 22:08:44.769643 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.769576    4477 scope.go:122] "RemoveContainer" containerID="c0c298ca2a98a892f23adecaddc83575f89fb6ed37cd952300aa86954ca586fb"
Oct 30 22:08:44.769805 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.769784    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0c298ca2a98a892f23adecaddc83575f89fb6ed37cd952300aa86954ca586fb"} err="failed to get container status \"c0c298ca2a98a892f23adecaddc83575f89fb6ed37cd952300aa86954ca586fb\": rpc error: code = NotFound desc = could not find container \"c0c298ca2a98a892f23adecaddc83575f89fb6ed37cd952300aa86954ca586fb\": container with ID starting with c0c298ca2a98a892f23adecaddc83575f89fb6ed37cd952300aa86954ca586fb not found: ID does not exist"
Oct 30 22:08:44.769805 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.769804    4477 scope.go:122] "RemoveContainer" containerID="e3cd27a71f050df532a08814a4280c80556187adb7c8d22fa82cd343369a30e0"
Oct 30 22:08:44.770024 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.770005    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3cd27a71f050df532a08814a4280c80556187adb7c8d22fa82cd343369a30e0"} err="failed to get container status \"e3cd27a71f050df532a08814a4280c80556187adb7c8d22fa82cd343369a30e0\": rpc error: code = NotFound desc = could not find container \"e3cd27a71f050df532a08814a4280c80556187adb7c8d22fa82cd343369a30e0\": container with ID starting with e3cd27a71f050df532a08814a4280c80556187adb7c8d22fa82cd343369a30e0 not found: ID does not exist"
Oct 30 22:08:44.770078 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.770024    4477 scope.go:122] "RemoveContainer" containerID="ab20b6db94f1e4cda42e0b032c7c382dea247c3c780f4399c70a7eb339cd66a6"
Oct 30 22:08:44.770253 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.770238    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab20b6db94f1e4cda42e0b032c7c382dea247c3c780f4399c70a7eb339cd66a6"} err="failed to get container status \"ab20b6db94f1e4cda42e0b032c7c382dea247c3c780f4399c70a7eb339cd66a6\": rpc error: code = NotFound desc = could not find container \"ab20b6db94f1e4cda42e0b032c7c382dea247c3c780f4399c70a7eb339cd66a6\": container with ID starting with ab20b6db94f1e4cda42e0b032c7c382dea247c3c780f4399c70a7eb339cd66a6 not found: ID does not exist"
Oct 30 22:08:44.776445 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.776426    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 22:08:44.786722 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.786701    4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 22:08:44.806724 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.806706    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 22:08:44.807135 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.807114    4477 cpu_manager.go:414] "containerMap: removing container" podUID="f904edd0-10dc-4958-8d9b-9e012c1319db" containerName="thanos-sidecar"
Oct 30 22:08:44.807197 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.807137    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f904edd0-10dc-4958-8d9b-9e012c1319db" containerName="thanos-sidecar"
Oct 30 22:08:44.807197 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.807157    4477 cpu_manager.go:414] "containerMap: removing container" podUID="f904edd0-10dc-4958-8d9b-9e012c1319db" containerName="config-reloader"
Oct 30 22:08:44.807197 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.807165    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f904edd0-10dc-4958-8d9b-9e012c1319db" containerName="config-reloader"
Oct 30 22:08:44.807197 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.807182    4477 cpu_manager.go:414] "containerMap: removing container" podUID="f904edd0-10dc-4958-8d9b-9e012c1319db" containerName="prometheus"
Oct 30 22:08:44.807197 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.807189    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f904edd0-10dc-4958-8d9b-9e012c1319db" containerName="prometheus"
Oct 30 22:08:44.807349 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.807200    4477 cpu_manager.go:414] "containerMap: removing container" podUID="f904edd0-10dc-4958-8d9b-9e012c1319db" containerName="init-config-reloader"
Oct 30 22:08:44.807349 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.807208    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f904edd0-10dc-4958-8d9b-9e012c1319db" containerName="init-config-reloader"
Oct 30 22:08:44.807349 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.807222    4477 cpu_manager.go:414] "containerMap: removing container" podUID="f904edd0-10dc-4958-8d9b-9e012c1319db" containerName="kube-rbac-proxy-web"
Oct 30 22:08:44.807349 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.807230    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f904edd0-10dc-4958-8d9b-9e012c1319db" containerName="kube-rbac-proxy-web"
Oct 30 22:08:44.807349 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.807259    4477 cpu_manager.go:414] "containerMap: removing container" podUID="f904edd0-10dc-4958-8d9b-9e012c1319db" containerName="kube-rbac-proxy"
Oct 30 22:08:44.807349 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.807266    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f904edd0-10dc-4958-8d9b-9e012c1319db" containerName="kube-rbac-proxy"
Oct 30 22:08:44.807349 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.807277    4477 cpu_manager.go:414] "containerMap: removing container" podUID="f904edd0-10dc-4958-8d9b-9e012c1319db" containerName="kube-rbac-proxy-thanos"
Oct 30 22:08:44.807349 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.807284    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f904edd0-10dc-4958-8d9b-9e012c1319db" containerName="kube-rbac-proxy-thanos"
Oct 30 22:08:44.807565 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.807358    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="f904edd0-10dc-4958-8d9b-9e012c1319db" containerName="kube-rbac-proxy"
Oct 30 22:08:44.807565 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.807369    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="f904edd0-10dc-4958-8d9b-9e012c1319db" containerName="kube-rbac-proxy-thanos"
Oct 30 22:08:44.807565 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.807378    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="f904edd0-10dc-4958-8d9b-9e012c1319db" containerName="thanos-sidecar"
Oct 30 22:08:44.807565 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.807389    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="f904edd0-10dc-4958-8d9b-9e012c1319db" containerName="config-reloader"
Oct 30 22:08:44.807565 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.807399    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="f904edd0-10dc-4958-8d9b-9e012c1319db" containerName="prometheus"
Oct 30 22:08:44.807565 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.807409    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="f904edd0-10dc-4958-8d9b-9e012c1319db" containerName="kube-rbac-proxy-web"
Oct 30 22:08:44.811565 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.811542    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:08:44.825868 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.825755    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Oct 30 22:08:44.825868 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.825769    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Oct 30 22:08:44.825868 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.825816    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Oct 30 22:08:44.825868 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.825845    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-26f0kjcpid9an\""
Oct 30 22:08:44.825868 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.825860    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-lghk9\""
Oct 30 22:08:44.826270 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.825875    4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Oct 30 22:08:44.826270 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.825892    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Oct 30 22:08:44.826270 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.825896    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Oct 30 22:08:44.826270 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.825925    4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Oct 30 22:08:44.826270 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.825927    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Oct 30 22:08:44.826270 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.825942    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Oct 30 22:08:44.826511 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.826286    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Oct 30 22:08:44.826511 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.826400    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Oct 30 22:08:44.826511 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.826453    4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Oct 30 22:08:44.838230 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.838206    4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Oct 30 22:08:44.838307 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.838292    4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Oct 30 22:08:44.850377 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.850357    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 22:08:44.856772 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.856749    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/030c1cfb-02a0-4628-9451-970fae0a8113-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"030c1cfb-02a0-4628-9451-970fae0a8113\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:08:44.856870 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.856784    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/030c1cfb-02a0-4628-9451-970fae0a8113-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"030c1cfb-02a0-4628-9451-970fae0a8113\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:08:44.856870 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.856813    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/030c1cfb-02a0-4628-9451-970fae0a8113-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"030c1cfb-02a0-4628-9451-970fae0a8113\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:08:44.856998 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.856885    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/030c1cfb-02a0-4628-9451-970fae0a8113-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"030c1cfb-02a0-4628-9451-970fae0a8113\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:08:44.857072 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.857023    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/030c1cfb-02a0-4628-9451-970fae0a8113-config\") pod \"prometheus-k8s-0\" (UID: \"030c1cfb-02a0-4628-9451-970fae0a8113\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:08:44.857123 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.857089    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/030c1cfb-02a0-4628-9451-970fae0a8113-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"030c1cfb-02a0-4628-9451-970fae0a8113\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:08:44.857168 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.857119    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/030c1cfb-02a0-4628-9451-970fae0a8113-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"030c1cfb-02a0-4628-9451-970fae0a8113\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:08:44.857168 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.857150    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/030c1cfb-02a0-4628-9451-970fae0a8113-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"030c1cfb-02a0-4628-9451-970fae0a8113\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:08:44.857338 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.857312    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/030c1cfb-02a0-4628-9451-970fae0a8113-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"030c1cfb-02a0-4628-9451-970fae0a8113\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:08:44.857411 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.857361    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/030c1cfb-02a0-4628-9451-970fae0a8113-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"030c1cfb-02a0-4628-9451-970fae0a8113\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:08:44.857466 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.857409    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/030c1cfb-02a0-4628-9451-970fae0a8113-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"030c1cfb-02a0-4628-9451-970fae0a8113\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:08:44.857466 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.857427    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/030c1cfb-02a0-4628-9451-970fae0a8113-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"030c1cfb-02a0-4628-9451-970fae0a8113\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:08:44.857466 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.857442    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/030c1cfb-02a0-4628-9451-970fae0a8113-web-config\") pod \"prometheus-k8s-0\" (UID: \"030c1cfb-02a0-4628-9451-970fae0a8113\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:08:44.857611 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.857471    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/030c1cfb-02a0-4628-9451-970fae0a8113-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"030c1cfb-02a0-4628-9451-970fae0a8113\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:08:44.857611 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.857493    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69hpd\" (UniqueName: \"kubernetes.io/projected/030c1cfb-02a0-4628-9451-970fae0a8113-kube-api-access-69hpd\") pod \"prometheus-k8s-0\" (UID: \"030c1cfb-02a0-4628-9451-970fae0a8113\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:08:44.857611 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.857512    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/030c1cfb-02a0-4628-9451-970fae0a8113-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"030c1cfb-02a0-4628-9451-970fae0a8113\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:08:44.857611 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.857540    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/030c1cfb-02a0-4628-9451-970fae0a8113-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"030c1cfb-02a0-4628-9451-970fae0a8113\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:08:44.857611 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.857569    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/030c1cfb-02a0-4628-9451-970fae0a8113-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"030c1cfb-02a0-4628-9451-970fae0a8113\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:08:44.857611 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.857587    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/030c1cfb-02a0-4628-9451-970fae0a8113-config-out\") pod \"prometheus-k8s-0\" (UID: \"030c1cfb-02a0-4628-9451-970fae0a8113\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:08:44.857611 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.857604    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/030c1cfb-02a0-4628-9451-970fae0a8113-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"030c1cfb-02a0-4628-9451-970fae0a8113\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:08:44.958633 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.958603    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/030c1cfb-02a0-4628-9451-970fae0a8113-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"030c1cfb-02a0-4628-9451-970fae0a8113\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:08:44.958633 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.958635    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/030c1cfb-02a0-4628-9451-970fae0a8113-config\") pod \"prometheus-k8s-0\" (UID: \"030c1cfb-02a0-4628-9451-970fae0a8113\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:08:44.958836 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.958658    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/030c1cfb-02a0-4628-9451-970fae0a8113-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"030c1cfb-02a0-4628-9451-970fae0a8113\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:08:44.958836 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.958677    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/030c1cfb-02a0-4628-9451-970fae0a8113-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"030c1cfb-02a0-4628-9451-970fae0a8113\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:08:44.958836 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.958739    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/030c1cfb-02a0-4628-9451-970fae0a8113-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"030c1cfb-02a0-4628-9451-970fae0a8113\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:08:44.958836 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.958776    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/030c1cfb-02a0-4628-9451-970fae0a8113-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"030c1cfb-02a0-4628-9451-970fae0a8113\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:08:44.958836 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.958801    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/030c1cfb-02a0-4628-9451-970fae0a8113-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"030c1cfb-02a0-4628-9451-970fae0a8113\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:08:44.958836 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.958829    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/030c1cfb-02a0-4628-9451-970fae0a8113-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"030c1cfb-02a0-4628-9451-970fae0a8113\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:08:44.959131 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.958858    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/030c1cfb-02a0-4628-9451-970fae0a8113-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"030c1cfb-02a0-4628-9451-970fae0a8113\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:08:44.959131 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.958881    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/030c1cfb-02a0-4628-9451-970fae0a8113-web-config\") pod \"prometheus-k8s-0\" (UID: \"030c1cfb-02a0-4628-9451-970fae0a8113\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:08:44.959219 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.959182    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/030c1cfb-02a0-4628-9451-970fae0a8113-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"030c1cfb-02a0-4628-9451-970fae0a8113\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:08:44.959278 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.959232    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69hpd\" (UniqueName: \"kubernetes.io/projected/030c1cfb-02a0-4628-9451-970fae0a8113-kube-api-access-69hpd\") pod \"prometheus-k8s-0\" (UID: \"030c1cfb-02a0-4628-9451-970fae0a8113\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:08:44.959330 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.959270    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/030c1cfb-02a0-4628-9451-970fae0a8113-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"030c1cfb-02a0-4628-9451-970fae0a8113\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:08:44.959330 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.959304    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/030c1cfb-02a0-4628-9451-970fae0a8113-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"030c1cfb-02a0-4628-9451-970fae0a8113\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:08:44.959426 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.959346    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/030c1cfb-02a0-4628-9451-970fae0a8113-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"030c1cfb-02a0-4628-9451-970fae0a8113\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:08:44.959426 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.959378    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/030c1cfb-02a0-4628-9451-970fae0a8113-config-out\") pod \"prometheus-k8s-0\" (UID: \"030c1cfb-02a0-4628-9451-970fae0a8113\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:08:44.959426 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.959402    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/030c1cfb-02a0-4628-9451-970fae0a8113-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"030c1cfb-02a0-4628-9451-970fae0a8113\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:08:44.959565 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.959430    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/030c1cfb-02a0-4628-9451-970fae0a8113-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"030c1cfb-02a0-4628-9451-970fae0a8113\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:08:44.959565 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.959441    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/030c1cfb-02a0-4628-9451-970fae0a8113-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"030c1cfb-02a0-4628-9451-970fae0a8113\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:08:44.959565 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.959462    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/030c1cfb-02a0-4628-9451-970fae0a8113-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"030c1cfb-02a0-4628-9451-970fae0a8113\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:08:44.959565 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.959494    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/030c1cfb-02a0-4628-9451-970fae0a8113-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"030c1cfb-02a0-4628-9451-970fae0a8113\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:08:44.960654 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.960204    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/030c1cfb-02a0-4628-9451-970fae0a8113-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"030c1cfb-02a0-4628-9451-970fae0a8113\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:08:44.960654 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.960204    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/030c1cfb-02a0-4628-9451-970fae0a8113-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"030c1cfb-02a0-4628-9451-970fae0a8113\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:08:44.960785 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.960699    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/030c1cfb-02a0-4628-9451-970fae0a8113-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"030c1cfb-02a0-4628-9451-970fae0a8113\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:08:44.961036 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.960981    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/030c1cfb-02a0-4628-9451-970fae0a8113-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"030c1cfb-02a0-4628-9451-970fae0a8113\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:08:44.961158 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.961034    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/030c1cfb-02a0-4628-9451-970fae0a8113-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"030c1cfb-02a0-4628-9451-970fae0a8113\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:08:44.961266 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.961240    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/030c1cfb-02a0-4628-9451-970fae0a8113-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"030c1cfb-02a0-4628-9451-970fae0a8113\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:08:44.961902 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.961873    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/030c1cfb-02a0-4628-9451-970fae0a8113-config\") pod \"prometheus-k8s-0\" (UID: \"030c1cfb-02a0-4628-9451-970fae0a8113\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:08:44.962790 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.962765    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/030c1cfb-02a0-4628-9451-970fae0a8113-config-out\") pod \"prometheus-k8s-0\" (UID: \"030c1cfb-02a0-4628-9451-970fae0a8113\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:08:44.963000 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.962948    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/030c1cfb-02a0-4628-9451-970fae0a8113-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"030c1cfb-02a0-4628-9451-970fae0a8113\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:08:44.963537 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.963340    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/030c1cfb-02a0-4628-9451-970fae0a8113-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"030c1cfb-02a0-4628-9451-970fae0a8113\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:08:44.963537 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.963446    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/030c1cfb-02a0-4628-9451-970fae0a8113-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"030c1cfb-02a0-4628-9451-970fae0a8113\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:08:44.963537 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.963490    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/030c1cfb-02a0-4628-9451-970fae0a8113-web-config\") pod \"prometheus-k8s-0\" (UID: \"030c1cfb-02a0-4628-9451-970fae0a8113\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:08:44.963829 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.963804    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/030c1cfb-02a0-4628-9451-970fae0a8113-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"030c1cfb-02a0-4628-9451-970fae0a8113\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:08:44.964117 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.964102    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/030c1cfb-02a0-4628-9451-970fae0a8113-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"030c1cfb-02a0-4628-9451-970fae0a8113\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:08:44.964833 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.964813    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/030c1cfb-02a0-4628-9451-970fae0a8113-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"030c1cfb-02a0-4628-9451-970fae0a8113\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:08:44.965126 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.965108    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/030c1cfb-02a0-4628-9451-970fae0a8113-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"030c1cfb-02a0-4628-9451-970fae0a8113\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:08:44.965215 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.965170    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/030c1cfb-02a0-4628-9451-970fae0a8113-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"030c1cfb-02a0-4628-9451-970fae0a8113\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:08:44.966109 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.966091    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/030c1cfb-02a0-4628-9451-970fae0a8113-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"030c1cfb-02a0-4628-9451-970fae0a8113\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:08:44.982721 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:44.982699    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69hpd\" (UniqueName: \"kubernetes.io/projected/030c1cfb-02a0-4628-9451-970fae0a8113-kube-api-access-69hpd\") pod \"prometheus-k8s-0\" (UID: \"030c1cfb-02a0-4628-9451-970fae0a8113\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:08:45.145118 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:45.145098    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:08:45.469526 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:45.469471    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 22:08:45.474791 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:08:45.474765    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod030c1cfb_02a0_4628_9451_970fae0a8113.slice/crio-4846b4b70be1b65bf85dadbc06d9262252a91fcca8281e1071998275afeb3d58 WatchSource:0}: Error finding container 4846b4b70be1b65bf85dadbc06d9262252a91fcca8281e1071998275afeb3d58: Status 404 returned error can't find the container with id 4846b4b70be1b65bf85dadbc06d9262252a91fcca8281e1071998275afeb3d58
Oct 30 22:08:45.719268 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:45.719238    4477 generic.go:356] "Generic (PLEG): container finished" podID="030c1cfb-02a0-4628-9451-970fae0a8113" containerID="976b768f3ee4201abe043c2ca27ebd60891df4a7a4650cdf3cadf8de97f079ba" exitCode=0
Oct 30 22:08:45.719373 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:45.719302    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"030c1cfb-02a0-4628-9451-970fae0a8113","Type":"ContainerDied","Data":"976b768f3ee4201abe043c2ca27ebd60891df4a7a4650cdf3cadf8de97f079ba"}
Oct 30 22:08:45.719428 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:45.719388    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"030c1cfb-02a0-4628-9451-970fae0a8113","Type":"ContainerStarted","Data":"4846b4b70be1b65bf85dadbc06d9262252a91fcca8281e1071998275afeb3d58"}
Oct 30 22:08:46.192320 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:46.192293    4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f904edd0-10dc-4958-8d9b-9e012c1319db" path="/var/lib/kubelet/pods/f904edd0-10dc-4958-8d9b-9e012c1319db/volumes"
Oct 30 22:08:46.731612 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:46.731565    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"030c1cfb-02a0-4628-9451-970fae0a8113","Type":"ContainerStarted","Data":"70563b93bf9708e2fae3d2e713e720856c04dbd6692fb138ea1d88f57d3cc886"}
Oct 30 22:08:46.731804 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:46.731652    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"030c1cfb-02a0-4628-9451-970fae0a8113","Type":"ContainerStarted","Data":"2792173fdf2e61eff121f26402680dca3a2bfd8cce803c20937a20f275800c9b"}
Oct 30 22:08:46.731804 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:46.731680    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"030c1cfb-02a0-4628-9451-970fae0a8113","Type":"ContainerStarted","Data":"39a3cecedd70c9318510943a1126a548e36fb612ec8fd6dc42173e6b9eed4f89"}
Oct 30 22:08:46.731804 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:46.731705    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"030c1cfb-02a0-4628-9451-970fae0a8113","Type":"ContainerStarted","Data":"99d6096adb364a5c74d4da8a78a74bde2d8751704682f07b3147610e42a5f8b9"}
Oct 30 22:08:46.731804 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:46.731750    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"030c1cfb-02a0-4628-9451-970fae0a8113","Type":"ContainerStarted","Data":"18303f48cd0facc330e90d44e0e719eba05b8364f02edf6ea96e478fbdb157e9"}
Oct 30 22:08:46.731804 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:46.731772    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"030c1cfb-02a0-4628-9451-970fae0a8113","Type":"ContainerStarted","Data":"41de40bd34362e85a0cbee3d8d8d3c131e660b2d31811bb02fd402f875573f75"}
Oct 30 22:08:46.812988 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:46.812940    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.7981001599999997 podStartE2EDuration="2.812927765s" totalImagesPullingTime="0s" totalInitContainerRuntime="14.827605ms" isStatefulPod=false podCreationTimestamp="2026-10-30 22:08:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:08:46.80190238 +0000 UTC m=+110.727828614" watchObservedRunningTime="2026-10-30 22:08:46.812927765 +0000 UTC m=+110.738853971"
Oct 30 22:08:50.146013 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:08:50.145980    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:08:56.264527 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:08:56.264493    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 22:09:26.266123 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:09:26.266077    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 22:09:45.145577 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:09:45.145540    4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:09:45.161444 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:09:45.161418    4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:09:45.971296 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:09:45.971267    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 22:09:56.266117 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:09:56.266080    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 22:10:26.264693 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:10:26.264660    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 22:10:56.265442 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:10:56.265367    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 22:11:26.265231 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:11:26.265191    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 22:11:56.206435 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:11:56.206409    4477 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jcf8l_015f218d-bf1f-4e7e-bb5d-b805af003246/ovn-acl-logging/0.log"
Oct 30 22:11:56.207550 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:11:56.207529    4477 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jcf8l_015f218d-bf1f-4e7e-bb5d-b805af003246/ovn-acl-logging/0.log"
Oct 30 22:11:56.264360 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:11:56.264327    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 22:11:57.712314 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:11:57.712279    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-c4c4484-qvj9s"]
Oct 30 22:11:57.715826 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:11:57.715803    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c4c4484-qvj9s"
Oct 30 22:11:57.775355 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:11:57.775329    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 30 22:11:57.775836 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:11:57.775820    4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Oct 30 22:11:57.776095 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:11:57.776081    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Oct 30 22:11:57.776630 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:11:57.776616    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 30 22:11:57.776739 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:11:57.776716    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-857vh\""
Oct 30 22:11:57.778177 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:11:57.778161    4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Oct 30 22:11:57.782538 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:11:57.782518    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/46782c7c-94be-495f-a34b-3e2fb6e821c1-webhook-cert\") pod \"opendatahub-operator-controller-manager-c4c4484-qvj9s\" (UID: \"46782c7c-94be-495f-a34b-3e2fb6e821c1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c4c4484-qvj9s"
Oct 30 22:11:57.782635 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:11:57.782542    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92gsv\" (UniqueName: \"kubernetes.io/projected/46782c7c-94be-495f-a34b-3e2fb6e821c1-kube-api-access-92gsv\") pod \"opendatahub-operator-controller-manager-c4c4484-qvj9s\" (UID: \"46782c7c-94be-495f-a34b-3e2fb6e821c1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c4c4484-qvj9s"
Oct 30 22:11:57.782635 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:11:57.782562    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/46782c7c-94be-495f-a34b-3e2fb6e821c1-manifests\") pod \"opendatahub-operator-controller-manager-c4c4484-qvj9s\" (UID: \"46782c7c-94be-495f-a34b-3e2fb6e821c1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c4c4484-qvj9s"
Oct 30 22:11:57.782635 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:11:57.782586    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/46782c7c-94be-495f-a34b-3e2fb6e821c1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-c4c4484-qvj9s\" (UID: \"46782c7c-94be-495f-a34b-3e2fb6e821c1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c4c4484-qvj9s"
Oct 30 22:11:57.782753 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:11:57.782721    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/46782c7c-94be-495f-a34b-3e2fb6e821c1-metrics-cert\") pod \"opendatahub-operator-controller-manager-c4c4484-qvj9s\" (UID: \"46782c7c-94be-495f-a34b-3e2fb6e821c1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c4c4484-qvj9s"
Oct 30 22:11:57.782753 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:11:57.782744    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/46782c7c-94be-495f-a34b-3e2fb6e821c1-tmp\") pod \"opendatahub-operator-controller-manager-c4c4484-qvj9s\" (UID: \"46782c7c-94be-495f-a34b-3e2fb6e821c1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c4c4484-qvj9s"
Oct 30 22:11:57.798110 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:11:57.796509    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-c4c4484-qvj9s"]
Oct 30 22:11:57.883717 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:11:57.883689    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/46782c7c-94be-495f-a34b-3e2fb6e821c1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-c4c4484-qvj9s\" (UID: \"46782c7c-94be-495f-a34b-3e2fb6e821c1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c4c4484-qvj9s"
Oct 30 22:11:57.883859 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:11:57.883729    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/46782c7c-94be-495f-a34b-3e2fb6e821c1-metrics-cert\") pod \"opendatahub-operator-controller-manager-c4c4484-qvj9s\" (UID: \"46782c7c-94be-495f-a34b-3e2fb6e821c1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c4c4484-qvj9s"
Oct 30 22:11:57.883859 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:11:57.883745    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/46782c7c-94be-495f-a34b-3e2fb6e821c1-tmp\") pod \"opendatahub-operator-controller-manager-c4c4484-qvj9s\" (UID: \"46782c7c-94be-495f-a34b-3e2fb6e821c1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c4c4484-qvj9s"
Oct 30 22:11:57.883859 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:11:57.883789    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/46782c7c-94be-495f-a34b-3e2fb6e821c1-webhook-cert\") pod \"opendatahub-operator-controller-manager-c4c4484-qvj9s\" (UID: \"46782c7c-94be-495f-a34b-3e2fb6e821c1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c4c4484-qvj9s"
Oct 30 22:11:57.883859 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:11:57.883805    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92gsv\" (UniqueName: \"kubernetes.io/projected/46782c7c-94be-495f-a34b-3e2fb6e821c1-kube-api-access-92gsv\") pod \"opendatahub-operator-controller-manager-c4c4484-qvj9s\" (UID: \"46782c7c-94be-495f-a34b-3e2fb6e821c1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c4c4484-qvj9s"
Oct 30 22:11:57.883859 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:11:57.883821    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/46782c7c-94be-495f-a34b-3e2fb6e821c1-manifests\") pod \"opendatahub-operator-controller-manager-c4c4484-qvj9s\" (UID: \"46782c7c-94be-495f-a34b-3e2fb6e821c1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c4c4484-qvj9s"
Oct 30 22:11:57.884250 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:11:57.884230    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/46782c7c-94be-495f-a34b-3e2fb6e821c1-manifests\") pod \"opendatahub-operator-controller-manager-c4c4484-qvj9s\" (UID: \"46782c7c-94be-495f-a34b-3e2fb6e821c1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c4c4484-qvj9s"
Oct 30 22:11:57.886261 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:11:57.886233    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/46782c7c-94be-495f-a34b-3e2fb6e821c1-tmp\") pod \"opendatahub-operator-controller-manager-c4c4484-qvj9s\" (UID: \"46782c7c-94be-495f-a34b-3e2fb6e821c1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c4c4484-qvj9s"
Oct 30 22:11:57.886453 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:11:57.886429    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/46782c7c-94be-495f-a34b-3e2fb6e821c1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-c4c4484-qvj9s\" (UID: \"46782c7c-94be-495f-a34b-3e2fb6e821c1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c4c4484-qvj9s"
Oct 30 22:11:57.886524 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:11:57.886437    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/46782c7c-94be-495f-a34b-3e2fb6e821c1-metrics-cert\") pod \"opendatahub-operator-controller-manager-c4c4484-qvj9s\" (UID: \"46782c7c-94be-495f-a34b-3e2fb6e821c1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c4c4484-qvj9s"
Oct 30 22:11:57.886524 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:11:57.886495    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/46782c7c-94be-495f-a34b-3e2fb6e821c1-webhook-cert\") pod \"opendatahub-operator-controller-manager-c4c4484-qvj9s\" (UID: \"46782c7c-94be-495f-a34b-3e2fb6e821c1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c4c4484-qvj9s"
Oct 30 22:11:57.907527 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:11:57.907497    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92gsv\" (UniqueName: \"kubernetes.io/projected/46782c7c-94be-495f-a34b-3e2fb6e821c1-kube-api-access-92gsv\") pod \"opendatahub-operator-controller-manager-c4c4484-qvj9s\" (UID: \"46782c7c-94be-495f-a34b-3e2fb6e821c1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c4c4484-qvj9s"
Oct 30 22:11:58.030205 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:11:58.030152    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c4c4484-qvj9s"
Oct 30 22:11:58.387578 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:11:58.387543    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-c4c4484-qvj9s"]
Oct 30 22:11:58.390627 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:11:58.390603    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46782c7c_94be_495f_a34b_3e2fb6e821c1.slice/crio-8e0522ac468d2c3309b12d1d5f040bd649c71219d2547703b3aa539c06aa9ee6 WatchSource:0}: Error finding container 8e0522ac468d2c3309b12d1d5f040bd649c71219d2547703b3aa539c06aa9ee6: Status 404 returned error can't find the container with id 8e0522ac468d2c3309b12d1d5f040bd649c71219d2547703b3aa539c06aa9ee6
Oct 30 22:11:58.392476 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:11:58.392454    4477 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 22:11:58.440520 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:11:58.440480    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c4c4484-qvj9s" event={"ID":"46782c7c-94be-495f-a34b-3e2fb6e821c1","Type":"ContainerStarted","Data":"8e0522ac468d2c3309b12d1d5f040bd649c71219d2547703b3aa539c06aa9ee6"}
Oct 30 22:12:01.462553 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:01.462515    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c4c4484-qvj9s" event={"ID":"46782c7c-94be-495f-a34b-3e2fb6e821c1","Type":"ContainerStarted","Data":"b0eff26ea2c10c056eceb8309299526fc71870cde6afbbaade9864ae2a04ab09"}
Oct 30 22:12:02.468289 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:02.468249    4477 generic.go:356] "Generic (PLEG): container finished" podID="46782c7c-94be-495f-a34b-3e2fb6e821c1" containerID="b0eff26ea2c10c056eceb8309299526fc71870cde6afbbaade9864ae2a04ab09" exitCode=0
Oct 30 22:12:02.468753 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:02.468286    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c4c4484-qvj9s" event={"ID":"46782c7c-94be-495f-a34b-3e2fb6e821c1","Type":"ContainerDied","Data":"b0eff26ea2c10c056eceb8309299526fc71870cde6afbbaade9864ae2a04ab09"}
Oct 30 22:12:02.774737 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:02.774662    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-ngccb"]
Oct 30 22:12:02.778296 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:02.778276    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-ngccb"
Oct 30 22:12:02.803490 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:02.803465    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-ngccb"]
Oct 30 22:12:02.833983 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:02.833957    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-gm9wj\""
Oct 30 22:12:02.834887 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:02.834871    4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Oct 30 22:12:02.835117 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:02.835101    4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Oct 30 22:12:02.931458 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:02.931435    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cd56ebbf-e82b-4db8-829e-cef149c3f35a-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-ngccb\" (UID: \"cd56ebbf-e82b-4db8-829e-cef149c3f35a\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-ngccb"
Oct 30 22:12:02.931563 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:02.931489    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68pkc\" (UniqueName: \"kubernetes.io/projected/cd56ebbf-e82b-4db8-829e-cef149c3f35a-kube-api-access-68pkc\") pod \"cert-manager-cainjector-5c57bbd4fc-ngccb\" (UID: \"cd56ebbf-e82b-4db8-829e-cef149c3f35a\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-ngccb"
Oct 30 22:12:03.032280 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:03.032217    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cd56ebbf-e82b-4db8-829e-cef149c3f35a-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-ngccb\" (UID: \"cd56ebbf-e82b-4db8-829e-cef149c3f35a\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-ngccb"
Oct 30 22:12:03.032280 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:03.032272    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68pkc\" (UniqueName: \"kubernetes.io/projected/cd56ebbf-e82b-4db8-829e-cef149c3f35a-kube-api-access-68pkc\") pod \"cert-manager-cainjector-5c57bbd4fc-ngccb\" (UID: \"cd56ebbf-e82b-4db8-829e-cef149c3f35a\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-ngccb"
Oct 30 22:12:03.057279 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:03.057250    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cd56ebbf-e82b-4db8-829e-cef149c3f35a-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-ngccb\" (UID: \"cd56ebbf-e82b-4db8-829e-cef149c3f35a\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-ngccb"
Oct 30 22:12:03.057393 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:03.057365    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68pkc\" (UniqueName: \"kubernetes.io/projected/cd56ebbf-e82b-4db8-829e-cef149c3f35a-kube-api-access-68pkc\") pod \"cert-manager-cainjector-5c57bbd4fc-ngccb\" (UID: \"cd56ebbf-e82b-4db8-829e-cef149c3f35a\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-ngccb"
Oct 30 22:12:03.106868 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:03.106847    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-ngccb"
Oct 30 22:12:03.399399 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:03.399339    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-ngccb"]
Oct 30 22:12:03.404137 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:12:03.404116    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcd56ebbf_e82b_4db8_829e_cef149c3f35a.slice/crio-e23c577457325c57e986f2c6df059b6c3a8ebf666aa560a4cc17879109d591f1 WatchSource:0}: Error finding container e23c577457325c57e986f2c6df059b6c3a8ebf666aa560a4cc17879109d591f1: Status 404 returned error can't find the container with id e23c577457325c57e986f2c6df059b6c3a8ebf666aa560a4cc17879109d591f1
Oct 30 22:12:03.473979 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:03.473946    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c4c4484-qvj9s" event={"ID":"46782c7c-94be-495f-a34b-3e2fb6e821c1","Type":"ContainerStarted","Data":"39b1e2a2e3a10853bcbe8fc6fc4ca88b7823764f0ed080fb4329f2942d23b5e8"}
Oct 30 22:12:03.474345 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:03.474090    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c4c4484-qvj9s"
Oct 30 22:12:03.475101 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:03.475072    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-ngccb" event={"ID":"cd56ebbf-e82b-4db8-829e-cef149c3f35a","Type":"ContainerStarted","Data":"e23c577457325c57e986f2c6df059b6c3a8ebf666aa560a4cc17879109d591f1"}
Oct 30 22:12:03.566149 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:03.566100    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c4c4484-qvj9s" podStartSLOduration=3.061072158 podStartE2EDuration="6.566087621s" totalImagesPullingTime="3.084854252s" totalInitContainerRuntime="420.161211ms" isStatefulPod=false podCreationTimestamp="2026-10-30 22:11:57 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:03.56237078 +0000 UTC m=+307.488296986" watchObservedRunningTime="2026-10-30 22:12:03.566087621 +0000 UTC m=+307.492013827"
Oct 30 22:12:06.078709 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:06.078676    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-fmvj2"]
Oct 30 22:12:06.082477 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:06.082450    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-fmvj2"
Oct 30 22:12:06.110354 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:06.110329    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-fmvj2"]
Oct 30 22:12:06.110489 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:06.110452    4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\""
Oct 30 22:12:06.110553 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:06.110544    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-dtg9w\""
Oct 30 22:12:06.110657 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:06.110642    4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\""
Oct 30 22:12:06.158773 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:06.158738    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4ckl\" (UniqueName: \"kubernetes.io/projected/79bcf22e-1311-464d-a06c-10548ee9067c-kube-api-access-l4ckl\") pod \"jobset-operator-674cd59dc5-fmvj2\" (UID: \"79bcf22e-1311-464d-a06c-10548ee9067c\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-fmvj2"
Oct 30 22:12:06.158894 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:06.158782    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/79bcf22e-1311-464d-a06c-10548ee9067c-tmp\") pod \"jobset-operator-674cd59dc5-fmvj2\" (UID: \"79bcf22e-1311-464d-a06c-10548ee9067c\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-fmvj2"
Oct 30 22:12:06.259255 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:06.259225    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4ckl\" (UniqueName: \"kubernetes.io/projected/79bcf22e-1311-464d-a06c-10548ee9067c-kube-api-access-l4ckl\") pod \"jobset-operator-674cd59dc5-fmvj2\" (UID: \"79bcf22e-1311-464d-a06c-10548ee9067c\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-fmvj2"
Oct 30 22:12:06.259434 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:06.259259    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/79bcf22e-1311-464d-a06c-10548ee9067c-tmp\") pod \"jobset-operator-674cd59dc5-fmvj2\" (UID: \"79bcf22e-1311-464d-a06c-10548ee9067c\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-fmvj2"
Oct 30 22:12:06.259546 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:06.259530    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/79bcf22e-1311-464d-a06c-10548ee9067c-tmp\") pod \"jobset-operator-674cd59dc5-fmvj2\" (UID: \"79bcf22e-1311-464d-a06c-10548ee9067c\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-fmvj2"
Oct 30 22:12:06.279787 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:06.279760    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4ckl\" (UniqueName: \"kubernetes.io/projected/79bcf22e-1311-464d-a06c-10548ee9067c-kube-api-access-l4ckl\") pod \"jobset-operator-674cd59dc5-fmvj2\" (UID: \"79bcf22e-1311-464d-a06c-10548ee9067c\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-fmvj2"
Oct 30 22:12:06.397118 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:06.397095    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-fmvj2"
Oct 30 22:12:06.493653 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:06.493623    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-ngccb" event={"ID":"cd56ebbf-e82b-4db8-829e-cef149c3f35a","Type":"ContainerStarted","Data":"1f903d9846923e791d4bb88f36c181854f5969a4e7e2195390fbe48149875e3b"}
Oct 30 22:12:06.542702 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:06.542539    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-ngccb" podStartSLOduration=1.6523712480000001 podStartE2EDuration="4.542521169s" totalImagesPullingTime="2.890149921s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:06.531846877 +0000 UTC m=+310.457773082" watchObservedRunningTime="2026-10-30 22:12:06.542521169 +0000 UTC m=+310.468447374"
Oct 30 22:12:06.697874 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:06.697799    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-fmvj2"]
Oct 30 22:12:06.702076 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:12:06.702034    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod79bcf22e_1311_464d_a06c_10548ee9067c.slice/crio-122dedce1c61f88ca3b8cf154f4a67325f44eb971a734cd7b15645271be06557 WatchSource:0}: Error finding container 122dedce1c61f88ca3b8cf154f4a67325f44eb971a734cd7b15645271be06557: Status 404 returned error can't find the container with id 122dedce1c61f88ca3b8cf154f4a67325f44eb971a734cd7b15645271be06557
Oct 30 22:12:07.500327 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:07.500285    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-fmvj2" event={"ID":"79bcf22e-1311-464d-a06c-10548ee9067c","Type":"ContainerStarted","Data":"122dedce1c61f88ca3b8cf154f4a67325f44eb971a734cd7b15645271be06557"}
Oct 30 22:12:10.518675 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:10.518624    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-fmvj2" event={"ID":"79bcf22e-1311-464d-a06c-10548ee9067c","Type":"ContainerStarted","Data":"829a3fedf8c8de0b8445dee71fdad538d260ace31739147c483b7a916202e581"}
Oct 30 22:12:10.557322 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:10.557274    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-fmvj2" podStartSLOduration=1.711315324 podStartE2EDuration="4.55726185s" totalImagesPullingTime="2.845946526s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:10.556225623 +0000 UTC m=+314.482151850" watchObservedRunningTime="2026-10-30 22:12:10.55726185 +0000 UTC m=+314.483188055"
Oct 30 22:12:14.484657 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:14.484606    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-c4c4484-qvj9s"
Oct 30 22:12:25.284453 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:25.284418    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-frfbm"]
Oct 30 22:12:25.288119 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:25.288094    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-frfbm"
Oct 30 22:12:25.304599 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:25.304579    4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 30 22:12:25.307428 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:25.307412    4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 30 22:12:25.313759 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:25.313738    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzmhw\" (UniqueName: \"kubernetes.io/projected/41c589fb-1f6a-442e-ad41-cd6a7ba1ae39-kube-api-access-pzmhw\") pod \"dns-operator-controller-manager-8557cbdb84-frfbm\" (UID: \"41c589fb-1f6a-442e-ad41-cd6a7ba1ae39\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-frfbm"
Oct 30 22:12:25.315701 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:25.315687    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-mth4r\""
Oct 30 22:12:25.320232 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:25.320213    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-frfbm"]
Oct 30 22:12:25.405645 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:25.405622    4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Oct 30 22:12:25.415119 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:25.415096    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzmhw\" (UniqueName: \"kubernetes.io/projected/41c589fb-1f6a-442e-ad41-cd6a7ba1ae39-kube-api-access-pzmhw\") pod \"dns-operator-controller-manager-8557cbdb84-frfbm\" (UID: \"41c589fb-1f6a-442e-ad41-cd6a7ba1ae39\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-frfbm"
Oct 30 22:12:25.433391 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:25.433369    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzmhw\" (UniqueName: \"kubernetes.io/projected/41c589fb-1f6a-442e-ad41-cd6a7ba1ae39-kube-api-access-pzmhw\") pod \"dns-operator-controller-manager-8557cbdb84-frfbm\" (UID: \"41c589fb-1f6a-442e-ad41-cd6a7ba1ae39\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-frfbm"
Oct 30 22:12:25.601453 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:25.601378    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-frfbm"
Oct 30 22:12:25.892544 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:25.892515    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-frfbm"]
Oct 30 22:12:25.896437 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:12:25.896412    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41c589fb_1f6a_442e_ad41_cd6a7ba1ae39.slice/crio-be4f4ef80c7bc1ad7e4241b73788325ef9030ef17804d1bb0fc673afe6dae1f0 WatchSource:0}: Error finding container be4f4ef80c7bc1ad7e4241b73788325ef9030ef17804d1bb0fc673afe6dae1f0: Status 404 returned error can't find the container with id be4f4ef80c7bc1ad7e4241b73788325ef9030ef17804d1bb0fc673afe6dae1f0
Oct 30 22:12:26.265918 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:12:26.265842    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 22:12:26.591556 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:26.591075    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-frfbm" event={"ID":"41c589fb-1f6a-442e-ad41-cd6a7ba1ae39","Type":"ContainerStarted","Data":"be4f4ef80c7bc1ad7e4241b73788325ef9030ef17804d1bb0fc673afe6dae1f0"}
Oct 30 22:12:28.190990 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:28.188132    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6f865f994d-vg2nd"]
Oct 30 22:12:28.192790 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:28.192765    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6f865f994d-vg2nd"
Oct 30 22:12:28.219892 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:28.219863    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Oct 30 22:12:28.220003 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:28.219923    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Oct 30 22:12:28.220003 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:28.219949    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-9nw5f\""
Oct 30 22:12:28.230990 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:28.230954    4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 22:12:28.231180 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:28.231033    4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 22:12:28.232977 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:28.232951    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6f865f994d-vg2nd"]
Oct 30 22:12:28.237482 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:28.237458    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5fc4bd88-0db5-4608-ab12-88364e90d0c9-tmp\") pod \"kube-auth-proxy-6f865f994d-vg2nd\" (UID: \"5fc4bd88-0db5-4608-ab12-88364e90d0c9\") " pod="openshift-ingress/kube-auth-proxy-6f865f994d-vg2nd"
Oct 30 22:12:28.237585 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:28.237517    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fc4bd88-0db5-4608-ab12-88364e90d0c9-tls-certs\") pod \"kube-auth-proxy-6f865f994d-vg2nd\" (UID: \"5fc4bd88-0db5-4608-ab12-88364e90d0c9\") " pod="openshift-ingress/kube-auth-proxy-6f865f994d-vg2nd"
Oct 30 22:12:28.237585 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:28.237552    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzsfl\" (UniqueName: \"kubernetes.io/projected/5fc4bd88-0db5-4608-ab12-88364e90d0c9-kube-api-access-gzsfl\") pod \"kube-auth-proxy-6f865f994d-vg2nd\" (UID: \"5fc4bd88-0db5-4608-ab12-88364e90d0c9\") " pod="openshift-ingress/kube-auth-proxy-6f865f994d-vg2nd"
Oct 30 22:12:28.338271 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:28.338217    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fc4bd88-0db5-4608-ab12-88364e90d0c9-tls-certs\") pod \"kube-auth-proxy-6f865f994d-vg2nd\" (UID: \"5fc4bd88-0db5-4608-ab12-88364e90d0c9\") " pod="openshift-ingress/kube-auth-proxy-6f865f994d-vg2nd"
Oct 30 22:12:28.338271 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:28.338253    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzsfl\" (UniqueName: \"kubernetes.io/projected/5fc4bd88-0db5-4608-ab12-88364e90d0c9-kube-api-access-gzsfl\") pod \"kube-auth-proxy-6f865f994d-vg2nd\" (UID: \"5fc4bd88-0db5-4608-ab12-88364e90d0c9\") " pod="openshift-ingress/kube-auth-proxy-6f865f994d-vg2nd"
Oct 30 22:12:28.338393 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:28.338297    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5fc4bd88-0db5-4608-ab12-88364e90d0c9-tmp\") pod \"kube-auth-proxy-6f865f994d-vg2nd\" (UID: \"5fc4bd88-0db5-4608-ab12-88364e90d0c9\") " pod="openshift-ingress/kube-auth-proxy-6f865f994d-vg2nd"
Oct 30 22:12:28.340432 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:28.340408    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5fc4bd88-0db5-4608-ab12-88364e90d0c9-tmp\") pod \"kube-auth-proxy-6f865f994d-vg2nd\" (UID: \"5fc4bd88-0db5-4608-ab12-88364e90d0c9\") " pod="openshift-ingress/kube-auth-proxy-6f865f994d-vg2nd"
Oct 30 22:12:28.340635 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:28.340618    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fc4bd88-0db5-4608-ab12-88364e90d0c9-tls-certs\") pod \"kube-auth-proxy-6f865f994d-vg2nd\" (UID: \"5fc4bd88-0db5-4608-ab12-88364e90d0c9\") " pod="openshift-ingress/kube-auth-proxy-6f865f994d-vg2nd"
Oct 30 22:12:28.371703 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:28.371679    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzsfl\" (UniqueName: \"kubernetes.io/projected/5fc4bd88-0db5-4608-ab12-88364e90d0c9-kube-api-access-gzsfl\") pod \"kube-auth-proxy-6f865f994d-vg2nd\" (UID: \"5fc4bd88-0db5-4608-ab12-88364e90d0c9\") " pod="openshift-ingress/kube-auth-proxy-6f865f994d-vg2nd"
Oct 30 22:12:28.505734 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:28.505704    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6f865f994d-vg2nd"
Oct 30 22:12:28.602153 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:28.602079    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-frfbm" event={"ID":"41c589fb-1f6a-442e-ad41-cd6a7ba1ae39","Type":"ContainerStarted","Data":"fcc6eca722bf4b6527367a1ff4a1b2ef156a2f18ee932e73af05d306a2ec40aa"}
Oct 30 22:12:28.602303 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:28.602228    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-frfbm"
Oct 30 22:12:28.646799 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:28.646756    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-frfbm" podStartSLOduration=1.510093242 podStartE2EDuration="3.646743768s" totalImagesPullingTime="2.136650526s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:28.645526093 +0000 UTC m=+332.571452319" watchObservedRunningTime="2026-10-30 22:12:28.646743768 +0000 UTC m=+332.572669974"
Oct 30 22:12:28.822762 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:28.822731    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6f865f994d-vg2nd"]
Oct 30 22:12:28.826910 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:12:28.826887    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fc4bd88_0db5_4608_ab12_88364e90d0c9.slice/crio-a957d884f68dd2c8bf1d7f4c70fb97d8a0c8b07a1b52e58d6c27c3dcdaafb0a6 WatchSource:0}: Error finding container a957d884f68dd2c8bf1d7f4c70fb97d8a0c8b07a1b52e58d6c27c3dcdaafb0a6: Status 404 returned error can't find the container with id a957d884f68dd2c8bf1d7f4c70fb97d8a0c8b07a1b52e58d6c27c3dcdaafb0a6
Oct 30 22:12:29.608268 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:29.608230    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6f865f994d-vg2nd" event={"ID":"5fc4bd88-0db5-4608-ab12-88364e90d0c9","Type":"ContainerStarted","Data":"a957d884f68dd2c8bf1d7f4c70fb97d8a0c8b07a1b52e58d6c27c3dcdaafb0a6"}
Oct 30 22:12:30.253925 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:30.252355    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-r827g"]
Oct 30 22:12:30.261412 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:30.261384    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-r827g"
Oct 30 22:12:30.277229 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:30.277161    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-r827g"]
Oct 30 22:12:30.286340 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:30.286315    4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Oct 30 22:12:30.305429 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:30.305407    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-brshk\""
Oct 30 22:12:30.327272 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:30.327248    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-mfvqt"]
Oct 30 22:12:30.331252 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:30.331233    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-mfvqt"
Oct 30 22:12:30.350763 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:30.350728    4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Oct 30 22:12:30.351632 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:30.351610    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-mfvqt"]
Oct 30 22:12:30.357755 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:30.357731    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/b398c8cf-bcf4-4e3a-98d2-bbf862919745-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-mfvqt\" (UID: \"b398c8cf-bcf4-4e3a-98d2-bbf862919745\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-mfvqt"
Oct 30 22:12:30.357854 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:30.357769    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b398c8cf-bcf4-4e3a-98d2-bbf862919745-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-mfvqt\" (UID: \"b398c8cf-bcf4-4e3a-98d2-bbf862919745\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-mfvqt"
Oct 30 22:12:30.357917 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:30.357873    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dslg\" (UniqueName: \"kubernetes.io/projected/b398c8cf-bcf4-4e3a-98d2-bbf862919745-kube-api-access-2dslg\") pod \"kuadrant-console-plugin-66c4fb575d-mfvqt\" (UID: \"b398c8cf-bcf4-4e3a-98d2-bbf862919745\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-mfvqt"
Oct 30 22:12:30.357971 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:30.357941    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/22a74a86-6dba-4ba4-b557-68fb7a657066-config-file\") pod \"limitador-limitador-74dfcfb894-r827g\" (UID: \"22a74a86-6dba-4ba4-b557-68fb7a657066\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-r827g"
Oct 30 22:12:30.358004 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:30.357994    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmtg6\" (UniqueName: \"kubernetes.io/projected/22a74a86-6dba-4ba4-b557-68fb7a657066-kube-api-access-vmtg6\") pod \"limitador-limitador-74dfcfb894-r827g\" (UID: \"22a74a86-6dba-4ba4-b557-68fb7a657066\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-r827g"
Oct 30 22:12:30.381580 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:30.381556    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Oct 30 22:12:30.387325 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:30.387298    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-r827g"]
Oct 30 22:12:30.458680 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:30.458644    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/b398c8cf-bcf4-4e3a-98d2-bbf862919745-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-mfvqt\" (UID: \"b398c8cf-bcf4-4e3a-98d2-bbf862919745\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-mfvqt"
Oct 30 22:12:30.458824 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:30.458703    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b398c8cf-bcf4-4e3a-98d2-bbf862919745-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-mfvqt\" (UID: \"b398c8cf-bcf4-4e3a-98d2-bbf862919745\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-mfvqt"
Oct 30 22:12:30.458824 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:30.458746    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dslg\" (UniqueName: \"kubernetes.io/projected/b398c8cf-bcf4-4e3a-98d2-bbf862919745-kube-api-access-2dslg\") pod \"kuadrant-console-plugin-66c4fb575d-mfvqt\" (UID: \"b398c8cf-bcf4-4e3a-98d2-bbf862919745\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-mfvqt"
Oct 30 22:12:30.458957 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:30.458862    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/22a74a86-6dba-4ba4-b557-68fb7a657066-config-file\") pod \"limitador-limitador-74dfcfb894-r827g\" (UID: \"22a74a86-6dba-4ba4-b557-68fb7a657066\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-r827g"
Oct 30 22:12:30.458957 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:30.458910    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmtg6\" (UniqueName: \"kubernetes.io/projected/22a74a86-6dba-4ba4-b557-68fb7a657066-kube-api-access-vmtg6\") pod \"limitador-limitador-74dfcfb894-r827g\" (UID: \"22a74a86-6dba-4ba4-b557-68fb7a657066\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-r827g"
Oct 30 22:12:30.459449 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:30.459427    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b398c8cf-bcf4-4e3a-98d2-bbf862919745-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-mfvqt\" (UID: \"b398c8cf-bcf4-4e3a-98d2-bbf862919745\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-mfvqt"
Oct 30 22:12:30.459587 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:30.459470    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/22a74a86-6dba-4ba4-b557-68fb7a657066-config-file\") pod \"limitador-limitador-74dfcfb894-r827g\" (UID: \"22a74a86-6dba-4ba4-b557-68fb7a657066\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-r827g"
Oct 30 22:12:30.461109 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:30.461085    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/b398c8cf-bcf4-4e3a-98d2-bbf862919745-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-mfvqt\" (UID: \"b398c8cf-bcf4-4e3a-98d2-bbf862919745\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-mfvqt"
Oct 30 22:12:30.477721 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:30.477697    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmtg6\" (UniqueName: \"kubernetes.io/projected/22a74a86-6dba-4ba4-b557-68fb7a657066-kube-api-access-vmtg6\") pod \"limitador-limitador-74dfcfb894-r827g\" (UID: \"22a74a86-6dba-4ba4-b557-68fb7a657066\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-r827g"
Oct 30 22:12:30.477721 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:30.477715    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dslg\" (UniqueName: \"kubernetes.io/projected/b398c8cf-bcf4-4e3a-98d2-bbf862919745-kube-api-access-2dslg\") pod \"kuadrant-console-plugin-66c4fb575d-mfvqt\" (UID: \"b398c8cf-bcf4-4e3a-98d2-bbf862919745\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-mfvqt"
Oct 30 22:12:30.576738 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:30.576667    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-r827g"
Oct 30 22:12:30.645520 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:30.645488    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-mfvqt"
Oct 30 22:12:30.879125 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:30.879089    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-r827g"]
Oct 30 22:12:30.883114 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:12:30.883090    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22a74a86_6dba_4ba4_b557_68fb7a657066.slice/crio-6f5e4cf26f6daaf20bea4b6060844374c525f5a6cfda869cd6abc649c3fc22a7 WatchSource:0}: Error finding container 6f5e4cf26f6daaf20bea4b6060844374c525f5a6cfda869cd6abc649c3fc22a7: Status 404 returned error can't find the container with id 6f5e4cf26f6daaf20bea4b6060844374c525f5a6cfda869cd6abc649c3fc22a7
Oct 30 22:12:30.950052 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:30.950017    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-mfvqt"]
Oct 30 22:12:30.954399 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:12:30.954371    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb398c8cf_bcf4_4e3a_98d2_bbf862919745.slice/crio-3f146401b2b3f82ae1949ae592783d098697b2910c1dbfba16e5d14b857c0953 WatchSource:0}: Error finding container 3f146401b2b3f82ae1949ae592783d098697b2910c1dbfba16e5d14b857c0953: Status 404 returned error can't find the container with id 3f146401b2b3f82ae1949ae592783d098697b2910c1dbfba16e5d14b857c0953
Oct 30 22:12:31.624272 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:31.624238    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6f865f994d-vg2nd" event={"ID":"5fc4bd88-0db5-4608-ab12-88364e90d0c9","Type":"ContainerStarted","Data":"e39706ae1d1f7d78e707df2a41069b7c6aae051b60e35bf2740a41b40477b6b6"}
Oct 30 22:12:31.627547 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:31.627465    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-r827g" event={"ID":"22a74a86-6dba-4ba4-b557-68fb7a657066","Type":"ContainerStarted","Data":"6f5e4cf26f6daaf20bea4b6060844374c525f5a6cfda869cd6abc649c3fc22a7"}
Oct 30 22:12:31.629509 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:31.629471    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-mfvqt" event={"ID":"b398c8cf-bcf4-4e3a-98d2-bbf862919745","Type":"ContainerStarted","Data":"3f146401b2b3f82ae1949ae592783d098697b2910c1dbfba16e5d14b857c0953"}
Oct 30 22:12:31.667241 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:31.667190    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6f865f994d-vg2nd" podStartSLOduration=1.325529717 podStartE2EDuration="3.667174498s" totalImagesPullingTime="2.341644781s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:31.666602433 +0000 UTC m=+335.592528640" watchObservedRunningTime="2026-10-30 22:12:31.667174498 +0000 UTC m=+335.593100705"
Oct 30 22:12:31.889192 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:31.888162    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-jx558"]
Oct 30 22:12:31.892635 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:31.892609    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6bz92"]
Oct 30 22:12:31.892954 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:31.892740    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-jx558"
Oct 30 22:12:31.896377 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:31.896356    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-6bz92"
Oct 30 22:12:31.909734 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:31.909717    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-jx558"]
Oct 30 22:12:31.920327 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:31.920310    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-xwk44\""
Oct 30 22:12:31.938097 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:31.938077    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6bz92"]
Oct 30 22:12:31.963489 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:31.963419    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6bz92"]
Oct 30 22:12:31.982035 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:31.981826    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b0d95e35-ac33-4203-a312-e7b072d8445a-config-file\") pod \"limitador-limitador-7c9b87d574-6bz92\" (UID: \"b0d95e35-ac33-4203-a312-e7b072d8445a\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-6bz92"
Oct 30 22:12:31.982035 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:31.981871    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znrm7\" (UniqueName: \"kubernetes.io/projected/afa19129-2247-41d0-bd7b-5cff3a9894d3-kube-api-access-znrm7\") pod \"authorino-56c6c9ddd5-jx558\" (UID: \"afa19129-2247-41d0-bd7b-5cff3a9894d3\") " pod="kuadrant-system/authorino-56c6c9ddd5-jx558"
Oct 30 22:12:31.982035 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:31.981897    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q68m8\" (UniqueName: \"kubernetes.io/projected/b0d95e35-ac33-4203-a312-e7b072d8445a-kube-api-access-q68m8\") pod \"limitador-limitador-7c9b87d574-6bz92\" (UID: \"b0d95e35-ac33-4203-a312-e7b072d8445a\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-6bz92"
Oct 30 22:12:32.082397 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:32.082369    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b0d95e35-ac33-4203-a312-e7b072d8445a-config-file\") pod \"limitador-limitador-7c9b87d574-6bz92\" (UID: \"b0d95e35-ac33-4203-a312-e7b072d8445a\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-6bz92"
Oct 30 22:12:32.082545 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:32.082416    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znrm7\" (UniqueName: \"kubernetes.io/projected/afa19129-2247-41d0-bd7b-5cff3a9894d3-kube-api-access-znrm7\") pod \"authorino-56c6c9ddd5-jx558\" (UID: \"afa19129-2247-41d0-bd7b-5cff3a9894d3\") " pod="kuadrant-system/authorino-56c6c9ddd5-jx558"
Oct 30 22:12:32.082545 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:32.082438    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q68m8\" (UniqueName: \"kubernetes.io/projected/b0d95e35-ac33-4203-a312-e7b072d8445a-kube-api-access-q68m8\") pod \"limitador-limitador-7c9b87d574-6bz92\" (UID: \"b0d95e35-ac33-4203-a312-e7b072d8445a\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-6bz92"
Oct 30 22:12:32.083108 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:32.083079    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/b0d95e35-ac33-4203-a312-e7b072d8445a-config-file\") pod \"limitador-limitador-7c9b87d574-6bz92\" (UID: \"b0d95e35-ac33-4203-a312-e7b072d8445a\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-6bz92"
Oct 30 22:12:32.103760 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:32.103735    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q68m8\" (UniqueName: \"kubernetes.io/projected/b0d95e35-ac33-4203-a312-e7b072d8445a-kube-api-access-q68m8\") pod \"limitador-limitador-7c9b87d574-6bz92\" (UID: \"b0d95e35-ac33-4203-a312-e7b072d8445a\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-6bz92"
Oct 30 22:12:32.103879 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:32.103859    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znrm7\" (UniqueName: \"kubernetes.io/projected/afa19129-2247-41d0-bd7b-5cff3a9894d3-kube-api-access-znrm7\") pod \"authorino-56c6c9ddd5-jx558\" (UID: \"afa19129-2247-41d0-bd7b-5cff3a9894d3\") " pod="kuadrant-system/authorino-56c6c9ddd5-jx558"
Oct 30 22:12:32.218327 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:32.218302    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-jx558"
Oct 30 22:12:32.225282 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:32.225259    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-6bz92"
Oct 30 22:12:32.521505 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:32.521440    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-jx558"]
Oct 30 22:12:32.524894 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:12:32.524864    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podafa19129_2247_41d0_bd7b_5cff3a9894d3.slice/crio-28565701ae8fabf74b18c20895dbebf1320d1a741779ff426b1f6a8d9dd8f207 WatchSource:0}: Error finding container 28565701ae8fabf74b18c20895dbebf1320d1a741779ff426b1f6a8d9dd8f207: Status 404 returned error can't find the container with id 28565701ae8fabf74b18c20895dbebf1320d1a741779ff426b1f6a8d9dd8f207
Oct 30 22:12:32.543664 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:32.543536    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6bz92"]
Oct 30 22:12:32.546516 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:12:32.546489    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0d95e35_ac33_4203_a312_e7b072d8445a.slice/crio-c93c2b86074187763a54487ea7e3edce7a9be25079ad23eeae276ff968b6b192 WatchSource:0}: Error finding container c93c2b86074187763a54487ea7e3edce7a9be25079ad23eeae276ff968b6b192: Status 404 returned error can't find the container with id c93c2b86074187763a54487ea7e3edce7a9be25079ad23eeae276ff968b6b192
Oct 30 22:12:32.635267 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:32.635229    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-jx558" event={"ID":"afa19129-2247-41d0-bd7b-5cff3a9894d3","Type":"ContainerStarted","Data":"28565701ae8fabf74b18c20895dbebf1320d1a741779ff426b1f6a8d9dd8f207"}
Oct 30 22:12:32.637100 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:32.637074    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-6bz92" event={"ID":"b0d95e35-ac33-4203-a312-e7b072d8445a","Type":"ContainerStarted","Data":"1ae8c20fd354ec241078f6323aecf99b7e78018fe48c4d61b919777d3401881f"}
Oct 30 22:12:32.637212 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:32.637136    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-6bz92"
Oct 30 22:12:32.637212 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:32.637161    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-6bz92" event={"ID":"b0d95e35-ac33-4203-a312-e7b072d8445a","Type":"ContainerStarted","Data":"c93c2b86074187763a54487ea7e3edce7a9be25079ad23eeae276ff968b6b192"}
Oct 30 22:12:32.638734 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:32.638712    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-r827g" event={"ID":"22a74a86-6dba-4ba4-b557-68fb7a657066","Type":"ContainerStarted","Data":"056a27fe3f1f94b23e2b3001ac77d66c5a6388bcc3f9d1d1d2b7ac6ed10ae728"}
Oct 30 22:12:32.638867 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:32.638815    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-r827g"
Oct 30 22:12:32.675176 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:32.675130    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-6bz92" podStartSLOduration=1.67511571 podStartE2EDuration="1.67511571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:32.673263151 +0000 UTC m=+336.599189383" watchObservedRunningTime="2026-10-30 22:12:32.67511571 +0000 UTC m=+336.601041912"
Oct 30 22:12:32.709155 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:32.709104    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-r827g" podStartSLOduration=1.448151142 podStartE2EDuration="2.709092254s" totalImagesPullingTime="1.260941112s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:32.708460743 +0000 UTC m=+336.634386949" watchObservedRunningTime="2026-10-30 22:12:32.709092254 +0000 UTC m=+336.635018460"
Oct 30 22:12:34.650003 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:34.649967    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-jx558" event={"ID":"afa19129-2247-41d0-bd7b-5cff3a9894d3","Type":"ContainerStarted","Data":"feeeb196f08616fef9c57478d2802cf14f7d9d89ec69489e134c34d603e0d6ac"}
Oct 30 22:12:34.685275 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:34.685222    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-jx558" podStartSLOduration=2.169492273 podStartE2EDuration="3.685208394s" totalImagesPullingTime="1.515716121s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:34.684126395 +0000 UTC m=+338.610052612" watchObservedRunningTime="2026-10-30 22:12:34.685208394 +0000 UTC m=+338.611134600"
Oct 30 22:12:36.668022 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:36.667986    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-mfvqt" event={"ID":"b398c8cf-bcf4-4e3a-98d2-bbf862919745","Type":"ContainerStarted","Data":"409a647ce94a781d3f7d49db4656622f8819500b2e2af23907223f0187d8c5b4"}
Oct 30 22:12:36.703996 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:36.703948    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-mfvqt" podStartSLOduration=1.360365177 podStartE2EDuration="6.703935656s" totalImagesPullingTime="5.343570479s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:36.703705355 +0000 UTC m=+340.629631561" watchObservedRunningTime="2026-10-30 22:12:36.703935656 +0000 UTC m=+340.629861872"
Oct 30 22:12:39.611131 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:39.611101    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-frfbm"
Oct 30 22:12:43.651475 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:43.651443    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-r827g"
Oct 30 22:12:43.651939 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:43.651543    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-6bz92"
Oct 30 22:12:43.761974 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:43.761941    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-r827g"]
Oct 30 22:12:43.762188 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:43.762132    4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-r827g" podUID="22a74a86-6dba-4ba4-b557-68fb7a657066" containerName="limitador" containerID="cri-o://056a27fe3f1f94b23e2b3001ac77d66c5a6388bcc3f9d1d1d2b7ac6ed10ae728" gracePeriod=30
Oct 30 22:12:44.302779 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:44.302758    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-r827g"
Oct 30 22:12:44.396847 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:44.396826    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22a74a86-6dba-4ba4-b557-68fb7a657066-kube-api-access-vmtg6\" (UniqueName: \"kubernetes.io/projected/22a74a86-6dba-4ba4-b557-68fb7a657066-kube-api-access-vmtg6\") pod \"22a74a86-6dba-4ba4-b557-68fb7a657066\" (UID: \"22a74a86-6dba-4ba4-b557-68fb7a657066\") "
Oct 30 22:12:44.397024 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:44.397002    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22a74a86-6dba-4ba4-b557-68fb7a657066-config-file\" (UniqueName: \"kubernetes.io/configmap/22a74a86-6dba-4ba4-b557-68fb7a657066-config-file\") pod \"22a74a86-6dba-4ba4-b557-68fb7a657066\" (UID: \"22a74a86-6dba-4ba4-b557-68fb7a657066\") "
Oct 30 22:12:44.397343 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:44.397322    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22a74a86-6dba-4ba4-b557-68fb7a657066-config-file" pod "22a74a86-6dba-4ba4-b557-68fb7a657066" (UID: "22a74a86-6dba-4ba4-b557-68fb7a657066"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:12:44.398979 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:44.398956    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22a74a86-6dba-4ba4-b557-68fb7a657066-kube-api-access-vmtg6" pod "22a74a86-6dba-4ba4-b557-68fb7a657066" (UID: "22a74a86-6dba-4ba4-b557-68fb7a657066"). InnerVolumeSpecName "kube-api-access-vmtg6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:12:44.498524 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:44.498489    4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmtg6\" (UniqueName: \"kubernetes.io/projected/22a74a86-6dba-4ba4-b557-68fb7a657066-kube-api-access-vmtg6\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:12:44.498524 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:44.498523    4477 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/22a74a86-6dba-4ba4-b557-68fb7a657066-config-file\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:12:44.704311 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:44.704252    4477 generic.go:356] "Generic (PLEG): container finished" podID="22a74a86-6dba-4ba4-b557-68fb7a657066" containerID="056a27fe3f1f94b23e2b3001ac77d66c5a6388bcc3f9d1d1d2b7ac6ed10ae728" exitCode=0
Oct 30 22:12:44.704311 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:44.704301    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-r827g"
Oct 30 22:12:44.704665 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:44.704337    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-r827g" event={"ID":"22a74a86-6dba-4ba4-b557-68fb7a657066","Type":"ContainerDied","Data":"056a27fe3f1f94b23e2b3001ac77d66c5a6388bcc3f9d1d1d2b7ac6ed10ae728"}
Oct 30 22:12:44.704665 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:44.704392    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-r827g" event={"ID":"22a74a86-6dba-4ba4-b557-68fb7a657066","Type":"ContainerDied","Data":"6f5e4cf26f6daaf20bea4b6060844374c525f5a6cfda869cd6abc649c3fc22a7"}
Oct 30 22:12:44.704665 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:44.704421    4477 scope.go:122] "RemoveContainer" containerID="056a27fe3f1f94b23e2b3001ac77d66c5a6388bcc3f9d1d1d2b7ac6ed10ae728"
Oct 30 22:12:44.713284 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:44.713267    4477 scope.go:122] "RemoveContainer" containerID="056a27fe3f1f94b23e2b3001ac77d66c5a6388bcc3f9d1d1d2b7ac6ed10ae728"
Oct 30 22:12:44.713551 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:12:44.713533    4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"056a27fe3f1f94b23e2b3001ac77d66c5a6388bcc3f9d1d1d2b7ac6ed10ae728\": container with ID starting with 056a27fe3f1f94b23e2b3001ac77d66c5a6388bcc3f9d1d1d2b7ac6ed10ae728 not found: ID does not exist" containerID="056a27fe3f1f94b23e2b3001ac77d66c5a6388bcc3f9d1d1d2b7ac6ed10ae728"
Oct 30 22:12:44.713613 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:44.713566    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"056a27fe3f1f94b23e2b3001ac77d66c5a6388bcc3f9d1d1d2b7ac6ed10ae728"} err="failed to get container status \"056a27fe3f1f94b23e2b3001ac77d66c5a6388bcc3f9d1d1d2b7ac6ed10ae728\": rpc error: code = NotFound desc = could not find container \"056a27fe3f1f94b23e2b3001ac77d66c5a6388bcc3f9d1d1d2b7ac6ed10ae728\": container with ID starting with 056a27fe3f1f94b23e2b3001ac77d66c5a6388bcc3f9d1d1d2b7ac6ed10ae728 not found: ID does not exist"
Oct 30 22:12:44.752239 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:44.752211    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-r827g"]
Oct 30 22:12:44.762594 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:44.762576    4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-r827g"]
Oct 30 22:12:46.192090 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:46.192059    4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22a74a86-6dba-4ba4-b557-68fb7a657066" path="/var/lib/kubelet/pods/22a74a86-6dba-4ba4-b557-68fb7a657066/volumes"
Oct 30 22:12:50.682063 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:50.682007    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-lx5nm"]
Oct 30 22:12:50.682683 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:50.682660    4477 cpu_manager.go:414] "containerMap: removing container" podUID="22a74a86-6dba-4ba4-b557-68fb7a657066" containerName="limitador"
Oct 30 22:12:50.682751 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:50.682687    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22a74a86-6dba-4ba4-b557-68fb7a657066" containerName="limitador"
Oct 30 22:12:50.682787 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:50.682780    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="22a74a86-6dba-4ba4-b557-68fb7a657066" containerName="limitador"
Oct 30 22:12:50.685835 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:50.685815    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-lx5nm"
Oct 30 22:12:50.707427 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:50.707394    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-lx5nm"]
Oct 30 22:12:50.725234 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:50.725210    4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 22:12:50.725410 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:50.725387    4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 22:12:50.725483 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:50.725461    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Oct 30 22:12:50.734784 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:50.734763    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-q47gm\""
Oct 30 22:12:50.752156 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:50.752128    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhwgq\" (UniqueName: \"kubernetes.io/projected/8a241719-733b-4527-85a9-727977fc863c-kube-api-access-dhwgq\") pod \"postgres-7cdbf6dc44-lx5nm\" (UID: \"8a241719-733b-4527-85a9-727977fc863c\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-lx5nm"
Oct 30 22:12:50.752251 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:50.752174    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/8a241719-733b-4527-85a9-727977fc863c-data\") pod \"postgres-7cdbf6dc44-lx5nm\" (UID: \"8a241719-733b-4527-85a9-727977fc863c\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-lx5nm"
Oct 30 22:12:50.853009 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:50.852982    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhwgq\" (UniqueName: \"kubernetes.io/projected/8a241719-733b-4527-85a9-727977fc863c-kube-api-access-dhwgq\") pod \"postgres-7cdbf6dc44-lx5nm\" (UID: \"8a241719-733b-4527-85a9-727977fc863c\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-lx5nm"
Oct 30 22:12:50.853152 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:50.853021    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/8a241719-733b-4527-85a9-727977fc863c-data\") pod \"postgres-7cdbf6dc44-lx5nm\" (UID: \"8a241719-733b-4527-85a9-727977fc863c\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-lx5nm"
Oct 30 22:12:50.853425 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:50.853403    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/8a241719-733b-4527-85a9-727977fc863c-data\") pod \"postgres-7cdbf6dc44-lx5nm\" (UID: \"8a241719-733b-4527-85a9-727977fc863c\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-lx5nm"
Oct 30 22:12:50.872051 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:50.872025    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhwgq\" (UniqueName: \"kubernetes.io/projected/8a241719-733b-4527-85a9-727977fc863c-kube-api-access-dhwgq\") pod \"postgres-7cdbf6dc44-lx5nm\" (UID: \"8a241719-733b-4527-85a9-727977fc863c\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-lx5nm"
Oct 30 22:12:50.998196 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:50.998133    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-lx5nm"
Oct 30 22:12:51.287905 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:51.287837    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-lx5nm"]
Oct 30 22:12:51.291440 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:12:51.291409    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a241719_733b_4527_85a9_727977fc863c.slice/crio-0c9cd38599acd3281edfc22d0aaa4ae48275569a8362be5706f139669c5d0599 WatchSource:0}: Error finding container 0c9cd38599acd3281edfc22d0aaa4ae48275569a8362be5706f139669c5d0599: Status 404 returned error can't find the container with id 0c9cd38599acd3281edfc22d0aaa4ae48275569a8362be5706f139669c5d0599
Oct 30 22:12:51.738289 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:51.738256    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-lx5nm" event={"ID":"8a241719-733b-4527-85a9-727977fc863c","Type":"ContainerStarted","Data":"0c9cd38599acd3281edfc22d0aaa4ae48275569a8362be5706f139669c5d0599"}
Oct 30 22:12:56.176378 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:56.176357    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Oct 30 22:12:56.267819 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:12:56.267789    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 22:12:56.770508 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:56.770472    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-lx5nm" event={"ID":"8a241719-733b-4527-85a9-727977fc863c","Type":"ContainerStarted","Data":"d9c6e650fcd00cf70656bd94ce55a0b1bbe89a0141e2906bc589164680df9305"}
Oct 30 22:12:56.770713 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:56.770535    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-lx5nm"
Oct 30 22:12:56.806614 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:12:56.806570    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-lx5nm" podStartSLOduration=1.935789004 podStartE2EDuration="6.806558181s" totalImagesPullingTime="4.870769177s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:56.806205474 +0000 UTC m=+360.732131680" watchObservedRunningTime="2026-10-30 22:12:56.806558181 +0000 UTC m=+360.732484386"
Oct 30 22:13:02.803828 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:13:02.803803    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-lx5nm"
Oct 30 22:13:15.377745 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:13:15.377708    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-x69f9"]
Oct 30 22:13:15.381223 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:13:15.381204    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-x69f9"
Oct 30 22:13:15.394735 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:13:15.394715    4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\""
Oct 30 22:13:15.394874 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:13:15.394765    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\""
Oct 30 22:13:15.394874 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:13:15.394826    4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 30 22:13:15.394874 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:13:15.394836    4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\""
Oct 30 22:13:15.405599 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:13:15.405554    4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 30 22:13:15.406072 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:13:15.405942    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-65t8j\""
Oct 30 22:13:15.408380 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:13:15.408360    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-x69f9"]
Oct 30 22:13:15.457935 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:13:15.457908    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/2d3c4dc5-460f-4d28-bad8-e97c462fe021-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-x69f9\" (UID: \"2d3c4dc5-460f-4d28-bad8-e97c462fe021\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-x69f9"
Oct 30 22:13:15.458110 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:13:15.457957    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2d3c4dc5-460f-4d28-bad8-e97c462fe021-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-x69f9\" (UID: \"2d3c4dc5-460f-4d28-bad8-e97c462fe021\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-x69f9"
Oct 30 22:13:15.458110 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:13:15.458091    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2d3c4dc5-460f-4d28-bad8-e97c462fe021-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-x69f9\" (UID: \"2d3c4dc5-460f-4d28-bad8-e97c462fe021\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-x69f9"
Oct 30 22:13:15.458230 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:13:15.458167    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvjv9\" (UniqueName: \"kubernetes.io/projected/2d3c4dc5-460f-4d28-bad8-e97c462fe021-kube-api-access-qvjv9\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-x69f9\" (UID: \"2d3c4dc5-460f-4d28-bad8-e97c462fe021\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-x69f9"
Oct 30 22:13:15.559163 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:13:15.559140    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvjv9\" (UniqueName: \"kubernetes.io/projected/2d3c4dc5-460f-4d28-bad8-e97c462fe021-kube-api-access-qvjv9\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-x69f9\" (UID: \"2d3c4dc5-460f-4d28-bad8-e97c462fe021\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-x69f9"
Oct 30 22:13:15.559300 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:13:15.559176    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/2d3c4dc5-460f-4d28-bad8-e97c462fe021-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-x69f9\" (UID: \"2d3c4dc5-460f-4d28-bad8-e97c462fe021\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-x69f9"
Oct 30 22:13:15.559300 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:13:15.559218    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2d3c4dc5-460f-4d28-bad8-e97c462fe021-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-x69f9\" (UID: \"2d3c4dc5-460f-4d28-bad8-e97c462fe021\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-x69f9"
Oct 30 22:13:15.559300 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:13:15.559255    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2d3c4dc5-460f-4d28-bad8-e97c462fe021-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-x69f9\" (UID: \"2d3c4dc5-460f-4d28-bad8-e97c462fe021\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-x69f9"
Oct 30 22:13:15.559916 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:13:15.559896    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2d3c4dc5-460f-4d28-bad8-e97c462fe021-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-x69f9\" (UID: \"2d3c4dc5-460f-4d28-bad8-e97c462fe021\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-x69f9"
Oct 30 22:13:15.560136 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:13:15.560117    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/2d3c4dc5-460f-4d28-bad8-e97c462fe021-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-x69f9\" (UID: \"2d3c4dc5-460f-4d28-bad8-e97c462fe021\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-x69f9"
Oct 30 22:13:15.561779 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:13:15.561757    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/2d3c4dc5-460f-4d28-bad8-e97c462fe021-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-x69f9\" (UID: \"2d3c4dc5-460f-4d28-bad8-e97c462fe021\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-x69f9"
Oct 30 22:13:15.576864 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:13:15.576838    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvjv9\" (UniqueName: \"kubernetes.io/projected/2d3c4dc5-460f-4d28-bad8-e97c462fe021-kube-api-access-qvjv9\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-x69f9\" (UID: \"2d3c4dc5-460f-4d28-bad8-e97c462fe021\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-x69f9"
Oct 30 22:13:15.693864 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:13:15.693813    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-x69f9"
Oct 30 22:13:15.986423 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:13:15.986341    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-x69f9"]
Oct 30 22:13:15.990388 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:13:15.990357    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d3c4dc5_460f_4d28_bad8_e97c462fe021.slice/crio-f59fce25e9a4d7e2f664d8a4a40de11dfeff06e66538e6c7dba03fc246808e61 WatchSource:0}: Error finding container f59fce25e9a4d7e2f664d8a4a40de11dfeff06e66538e6c7dba03fc246808e61: Status 404 returned error can't find the container with id f59fce25e9a4d7e2f664d8a4a40de11dfeff06e66538e6c7dba03fc246808e61
Oct 30 22:13:16.860703 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:13:16.860280    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-x69f9" event={"ID":"2d3c4dc5-460f-4d28-bad8-e97c462fe021","Type":"ContainerStarted","Data":"f59fce25e9a4d7e2f664d8a4a40de11dfeff06e66538e6c7dba03fc246808e61"}
Oct 30 22:13:26.265713 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:13:26.265686    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 22:13:26.914451 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:13:26.914414    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-x69f9" event={"ID":"2d3c4dc5-460f-4d28-bad8-e97c462fe021","Type":"ContainerStarted","Data":"e4c1efe4600a05f9d5a2eb11f9d4fe6fe423a9ec913ef87e831a61a413f0a98c"}
Oct 30 22:13:26.914612 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:13:26.914598    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-x69f9"
Oct 30 22:13:26.955841 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:13:26.955711    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-x69f9" podStartSLOduration=1.693914294 podStartE2EDuration="11.955698452s" totalImagesPullingTime="10.261784158s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:13:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:13:26.953752676 +0000 UTC m=+390.879678881" watchObservedRunningTime="2026-10-30 22:13:26.955698452 +0000 UTC m=+390.881624657"
Oct 30 22:13:27.701823 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:13:27.701787    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-ddzfb"]
Oct 30 22:13:27.705630 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:13:27.705614    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-ddzfb"
Oct 30 22:13:27.727635 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:13:27.727607    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-ddzfb"]
Oct 30 22:13:27.737836 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:13:27.737812    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-x68fm"]
Oct 30 22:13:27.741830 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:13:27.741811    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-x68fm"
Oct 30 22:13:27.749105 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:13:27.749085    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Oct 30 22:13:27.751480 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:13:27.751457    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-tzljq\""
Oct 30 22:13:27.755654 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:13:27.755637    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-6ktn8\""
Oct 30 22:13:27.755738 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:13:27.755660    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Oct 30 22:13:27.760246 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:13:27.760226    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-x68fm"]
Oct 30 22:13:27.780470 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:13:27.780446    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-f29pm"]
Oct 30 22:13:27.784194 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:13:27.784175    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-f29pm"
Oct 30 22:13:27.790665 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:13:27.790642    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pz6j9\" (UniqueName: \"kubernetes.io/projected/95aafcdd-d16c-40af-8dd3-e0fcefcbf5f9-kube-api-access-pz6j9\") pod \"kserve-controller-manager-776cc66d47-ddzfb\" (UID: \"95aafcdd-d16c-40af-8dd3-e0fcefcbf5f9\") " pod="opendatahub/kserve-controller-manager-776cc66d47-ddzfb"
Oct 30 22:13:27.790779 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:13:27.790738    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfq8d\" (UniqueName: \"kubernetes.io/projected/93a707d7-2921-438d-914b-193b1736f4f9-kube-api-access-pfq8d\") pod \"llmisvc-controller-manager-869d5dcc5d-x68fm\" (UID: \"93a707d7-2921-438d-914b-193b1736f4f9\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-x68fm"
Oct 30 22:13:27.790840 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:13:27.790774    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/95aafcdd-d16c-40af-8dd3-e0fcefcbf5f9-cert\") pod \"kserve-controller-manager-776cc66d47-ddzfb\" (UID: \"95aafcdd-d16c-40af-8dd3-e0fcefcbf5f9\") " pod="opendatahub/kserve-controller-manager-776cc66d47-ddzfb"
Oct 30 22:13:27.790897 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:13:27.790870    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/93a707d7-2921-438d-914b-193b1736f4f9-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-x68fm\" (UID: \"93a707d7-2921-438d-914b-193b1736f4f9\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-x68fm"
Oct 30 22:13:27.806904 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:13:27.806885    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-f29pm"]
Oct 30 22:13:27.821053 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:13:27.821020    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-ssczl"]
Oct 30 22:13:27.824819 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:13:27.824787    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-92crr\""
Oct 30 22:13:27.824819 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:13:27.824792    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Oct 30 22:13:27.824937 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:13:27.824825    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-ssczl"
Oct 30 22:13:27.844841 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:13:27.844817    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-ssczl"]
Oct 30 22:13:27.851884 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:13:27.851867    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\""
Oct 30 22:13:27.852065 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:13:27.852028    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Oct 30 22:13:27.852157 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:13:27.852090    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-g64qg\""
Oct 30 22:13:27.892321 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:13:27.892302    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pz6j9\" (UniqueName: \"kubernetes.io/projected/95aafcdd-d16c-40af-8dd3-e0fcefcbf5f9-kube-api-access-pz6j9\") pod \"kserve-controller-manager-776cc66d47-ddzfb\" (UID: \"95aafcdd-d16c-40af-8dd3-e0fcefcbf5f9\") " pod="opendatahub/kserve-controller-manager-776cc66d47-ddzfb"
Oct 30 22:13:27.892419 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:13:27.892346    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d33baf4-42cd-409c-a3fe-8a0bfcf7b1c5-tls-certs\") pod \"model-serving-api-75bd89fdfc-f29pm\" (UID: \"1d33baf4-42cd-409c-a3fe-8a0bfcf7b1c5\") " pod="opendatahub/model-serving-api-75bd89fdfc-f29pm"
Oct 30 22:13:27.892419 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:13:27.892363    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pjqz\" (UniqueName: \"kubernetes.io/projected/1d33baf4-42cd-409c-a3fe-8a0bfcf7b1c5-kube-api-access-7pjqz\") pod \"model-serving-api-75bd89fdfc-f29pm\" (UID: \"1d33baf4-42cd-409c-a3fe-8a0bfcf7b1c5\") " pod="opendatahub/model-serving-api-75bd89fdfc-f29pm"
Oct 30 22:13:27.892419 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:13:27.892382    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/12bab827-4026-4cfb-8182-cb686e075e8c-cert\") pod \"odh-model-controller-769875db56-ssczl\" (UID: \"12bab827-4026-4cfb-8182-cb686e075e8c\") " pod="opendatahub/odh-model-controller-769875db56-ssczl"
Oct 30 22:13:27.892527 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:13:27.892440    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jz5q5\" (UniqueName: \"kubernetes.io/projected/12bab827-4026-4cfb-8182-cb686e075e8c-kube-api-access-jz5q5\") pod \"odh-model-controller-769875db56-ssczl\" (UID: \"12bab827-4026-4cfb-8182-cb686e075e8c\") " pod="opendatahub/odh-model-controller-769875db56-ssczl"
Oct 30 22:13:27.892567 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:13:27.892528    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfq8d\" (UniqueName: \"kubernetes.io/projected/93a707d7-2921-438d-914b-193b1736f4f9-kube-api-access-pfq8d\") pod \"llmisvc-controller-manager-869d5dcc5d-x68fm\" (UID: \"93a707d7-2921-438d-914b-193b1736f4f9\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-x68fm"
Oct 30 22:13:27.892567 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:13:27.892549    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/95aafcdd-d16c-40af-8dd3-e0fcefcbf5f9-cert\") pod \"kserve-controller-manager-776cc66d47-ddzfb\" (UID: \"95aafcdd-d16c-40af-8dd3-e0fcefcbf5f9\") " pod="opendatahub/kserve-controller-manager-776cc66d47-ddzfb"
Oct 30 22:13:27.892635 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:13:27.892577    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12bab827-4026-4cfb-8182-cb686e075e8c-metrics-certs\") pod \"odh-model-controller-769875db56-ssczl\" (UID: \"12bab827-4026-4cfb-8182-cb686e075e8c\") " pod="opendatahub/odh-model-controller-769875db56-ssczl"
Oct 30 22:13:27.892635 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:13:27.892602    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/93a707d7-2921-438d-914b-193b1736f4f9-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-x68fm\" (UID: \"93a707d7-2921-438d-914b-193b1736f4f9\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-x68fm"
Oct 30 22:13:27.894933 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:13:27.894912    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/93a707d7-2921-438d-914b-193b1736f4f9-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-x68fm\" (UID: \"93a707d7-2921-438d-914b-193b1736f4f9\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-x68fm"
Oct 30 22:13:27.895118 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:13:27.895095    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/95aafcdd-d16c-40af-8dd3-e0fcefcbf5f9-cert\") pod \"kserve-controller-manager-776cc66d47-ddzfb\" (UID: \"95aafcdd-d16c-40af-8dd3-e0fcefcbf5f9\") " pod="opendatahub/kserve-controller-manager-776cc66d47-ddzfb"
Oct 30 22:13:27.909700 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:13:27.909677    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pz6j9\" (UniqueName: \"kubernetes.io/projected/95aafcdd-d16c-40af-8dd3-e0fcefcbf5f9-kube-api-access-pz6j9\") pod \"kserve-controller-manager-776cc66d47-ddzfb\" (UID: \"95aafcdd-d16c-40af-8dd3-e0fcefcbf5f9\") " pod="opendatahub/kserve-controller-manager-776cc66d47-ddzfb"
Oct 30 22:13:27.910423 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:13:27.910404    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfq8d\" (UniqueName: \"kubernetes.io/projected/93a707d7-2921-438d-914b-193b1736f4f9-kube-api-access-pfq8d\") pod \"llmisvc-controller-manager-869d5dcc5d-x68fm\" (UID: \"93a707d7-2921-438d-914b-193b1736f4f9\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-x68fm"
Oct 30 22:13:27.993657 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:13:27.993601    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d33baf4-42cd-409c-a3fe-8a0bfcf7b1c5-tls-certs\") pod \"model-serving-api-75bd89fdfc-f29pm\" (UID: \"1d33baf4-42cd-409c-a3fe-8a0bfcf7b1c5\") " pod="opendatahub/model-serving-api-75bd89fdfc-f29pm"
Oct 30 22:13:27.993657 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:13:27.993630    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pjqz\" (UniqueName: \"kubernetes.io/projected/1d33baf4-42cd-409c-a3fe-8a0bfcf7b1c5-kube-api-access-7pjqz\") pod \"model-serving-api-75bd89fdfc-f29pm\" (UID: \"1d33baf4-42cd-409c-a3fe-8a0bfcf7b1c5\") " pod="opendatahub/model-serving-api-75bd89fdfc-f29pm"
Oct 30 22:13:27.993657 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:13:27.993646    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/12bab827-4026-4cfb-8182-cb686e075e8c-cert\") pod \"odh-model-controller-769875db56-ssczl\" (UID: \"12bab827-4026-4cfb-8182-cb686e075e8c\") " pod="opendatahub/odh-model-controller-769875db56-ssczl"
Oct 30 22:13:27.993847 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:13:27.993665    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jz5q5\" (UniqueName: \"kubernetes.io/projected/12bab827-4026-4cfb-8182-cb686e075e8c-kube-api-access-jz5q5\") pod \"odh-model-controller-769875db56-ssczl\" (UID: \"12bab827-4026-4cfb-8182-cb686e075e8c\") " pod="opendatahub/odh-model-controller-769875db56-ssczl"
Oct 30 22:13:27.993847 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:13:27.993776    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12bab827-4026-4cfb-8182-cb686e075e8c-metrics-certs\") pod \"odh-model-controller-769875db56-ssczl\" (UID: \"12bab827-4026-4cfb-8182-cb686e075e8c\") " pod="opendatahub/odh-model-controller-769875db56-ssczl"
Oct 30 22:13:27.995986 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:13:27.995965    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d33baf4-42cd-409c-a3fe-8a0bfcf7b1c5-tls-certs\") pod \"model-serving-api-75bd89fdfc-f29pm\" (UID: \"1d33baf4-42cd-409c-a3fe-8a0bfcf7b1c5\") " pod="opendatahub/model-serving-api-75bd89fdfc-f29pm"
Oct 30 22:13:27.996752 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:13:27.996193    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/12bab827-4026-4cfb-8182-cb686e075e8c-metrics-certs\") pod \"odh-model-controller-769875db56-ssczl\" (UID: \"12bab827-4026-4cfb-8182-cb686e075e8c\") " pod="opendatahub/odh-model-controller-769875db56-ssczl"
Oct 30 22:13:27.996752 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:13:27.996246    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/12bab827-4026-4cfb-8182-cb686e075e8c-cert\") pod \"odh-model-controller-769875db56-ssczl\" (UID: \"12bab827-4026-4cfb-8182-cb686e075e8c\") " pod="opendatahub/odh-model-controller-769875db56-ssczl"
Oct 30 22:13:28.011510 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:13:28.011488    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pjqz\" (UniqueName: \"kubernetes.io/projected/1d33baf4-42cd-409c-a3fe-8a0bfcf7b1c5-kube-api-access-7pjqz\") pod \"model-serving-api-75bd89fdfc-f29pm\" (UID: \"1d33baf4-42cd-409c-a3fe-8a0bfcf7b1c5\") " pod="opendatahub/model-serving-api-75bd89fdfc-f29pm"
Oct 30 22:13:28.011592 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:13:28.011573    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jz5q5\" (UniqueName: \"kubernetes.io/projected/12bab827-4026-4cfb-8182-cb686e075e8c-kube-api-access-jz5q5\") pod \"odh-model-controller-769875db56-ssczl\" (UID: \"12bab827-4026-4cfb-8182-cb686e075e8c\") " pod="opendatahub/odh-model-controller-769875db56-ssczl"
Oct 30 22:13:28.018416 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:13:28.018402    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-ddzfb"
Oct 30 22:13:28.054631 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:13:28.054603    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-x68fm"
Oct 30 22:13:28.101108 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:13:28.101077    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-f29pm"
Oct 30 22:13:28.138696 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:13:28.138480    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-ssczl"
Oct 30 22:13:28.310659 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:13:28.310588    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-ddzfb"]
Oct 30 22:13:28.314052 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:13:28.314013    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95aafcdd_d16c_40af_8dd3_e0fcefcbf5f9.slice/crio-c021c889273f9f3b2deefb8ac706e2b2d9c49e45424ea9c83a137d380ba86cd0 WatchSource:0}: Error finding container c021c889273f9f3b2deefb8ac706e2b2d9c49e45424ea9c83a137d380ba86cd0: Status 404 returned error can't find the container with id c021c889273f9f3b2deefb8ac706e2b2d9c49e45424ea9c83a137d380ba86cd0
Oct 30 22:13:28.356345 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:13:28.356288    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-x68fm"]
Oct 30 22:13:28.361598 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:13:28.361565    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod93a707d7_2921_438d_914b_193b1736f4f9.slice/crio-51277fdd8e24365aee73e001c4095453add5419b288ad54e745a291ad91beb1f WatchSource:0}: Error finding container 51277fdd8e24365aee73e001c4095453add5419b288ad54e745a291ad91beb1f: Status 404 returned error can't find the container with id 51277fdd8e24365aee73e001c4095453add5419b288ad54e745a291ad91beb1f
Oct 30 22:13:28.399879 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:13:28.399854    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-f29pm"]
Oct 30 22:13:28.403521 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:13:28.403498    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d33baf4_42cd_409c_a3fe_8a0bfcf7b1c5.slice/crio-10b79102d8621cf6b2d73d4f8e000dc1bec9f2f9a7aa2f4b6f63424ece34bd8f WatchSource:0}: Error finding container 10b79102d8621cf6b2d73d4f8e000dc1bec9f2f9a7aa2f4b6f63424ece34bd8f: Status 404 returned error can't find the container with id 10b79102d8621cf6b2d73d4f8e000dc1bec9f2f9a7aa2f4b6f63424ece34bd8f
Oct 30 22:13:28.444311 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:13:28.444266    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-ssczl"]
Oct 30 22:13:28.451809 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:13:28.451781    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12bab827_4026_4cfb_8182_cb686e075e8c.slice/crio-9b8ce279eab58b10c25a16d19543df975e1c9a04a3779abebc78c42b661ab4cb WatchSource:0}: Error finding container 9b8ce279eab58b10c25a16d19543df975e1c9a04a3779abebc78c42b661ab4cb: Status 404 returned error can't find the container with id 9b8ce279eab58b10c25a16d19543df975e1c9a04a3779abebc78c42b661ab4cb
Oct 30 22:13:28.932792 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:13:28.932717    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-ssczl" event={"ID":"12bab827-4026-4cfb-8182-cb686e075e8c","Type":"ContainerStarted","Data":"9b8ce279eab58b10c25a16d19543df975e1c9a04a3779abebc78c42b661ab4cb"}
Oct 30 22:13:28.934760 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:13:28.934690    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-f29pm" event={"ID":"1d33baf4-42cd-409c-a3fe-8a0bfcf7b1c5","Type":"ContainerStarted","Data":"10b79102d8621cf6b2d73d4f8e000dc1bec9f2f9a7aa2f4b6f63424ece34bd8f"}
Oct 30 22:13:28.943848 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:13:28.943813    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-ddzfb" event={"ID":"95aafcdd-d16c-40af-8dd3-e0fcefcbf5f9","Type":"ContainerStarted","Data":"c021c889273f9f3b2deefb8ac706e2b2d9c49e45424ea9c83a137d380ba86cd0"}
Oct 30 22:13:28.946886 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:13:28.946860    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-x68fm" event={"ID":"93a707d7-2921-438d-914b-193b1736f4f9","Type":"ContainerStarted","Data":"51277fdd8e24365aee73e001c4095453add5419b288ad54e745a291ad91beb1f"}
Oct 30 22:13:31.966695 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:13:31.966658    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-ssczl" event={"ID":"12bab827-4026-4cfb-8182-cb686e075e8c","Type":"ContainerStarted","Data":"499147d9379b504f2494ea95d404cfe15cde585273a7e841fa4cacc148981d0f"}
Oct 30 22:13:31.967134 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:13:31.966764    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-ssczl"
Oct 30 22:13:31.968266 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:13:31.968235    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-f29pm" event={"ID":"1d33baf4-42cd-409c-a3fe-8a0bfcf7b1c5","Type":"ContainerStarted","Data":"d62d21187a15e6adbda227b260a1eb5dd3d90f42d791cd05d875b4525e6b60ae"}
Oct 30 22:13:31.968389 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:13:31.968286    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-f29pm"
Oct 30 22:13:31.970169 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:13:31.970096    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-ddzfb" event={"ID":"95aafcdd-d16c-40af-8dd3-e0fcefcbf5f9","Type":"ContainerStarted","Data":"eef5c0353d0e802b54c24eafe168f70edd9a25e750312049526ca66ca9646ded"}
Oct 30 22:13:31.970286 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:13:31.970201    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-ddzfb"
Oct 30 22:13:31.973814 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:13:31.973788    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-x68fm" event={"ID":"93a707d7-2921-438d-914b-193b1736f4f9","Type":"ContainerStarted","Data":"70a16fb6de6b57b7a80386de0412712cffc5e31245e9c84bae405a8a2ad25545"}
Oct 30 22:13:31.973901 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:13:31.973878    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-x68fm"
Oct 30 22:13:32.003267 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:13:32.003222    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-ssczl" podStartSLOduration=1.615667864 podStartE2EDuration="5.003207174s" totalImagesPullingTime="3.38753931s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:13:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:13:32.001829335 +0000 UTC m=+395.927755560" watchObservedRunningTime="2026-10-30 22:13:32.003207174 +0000 UTC m=+395.929133379"
Oct 30 22:13:32.039173 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:13:32.039125    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-f29pm" podStartSLOduration=2.541160342 podStartE2EDuration="5.039113575s" totalImagesPullingTime="2.497953233s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:13:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:13:32.037101132 +0000 UTC m=+395.963027337" watchObservedRunningTime="2026-10-30 22:13:32.039113575 +0000 UTC m=+395.965039781"
Oct 30 22:13:32.074766 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:13:32.074724    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-ddzfb" podStartSLOduration=2.000733634 podStartE2EDuration="5.074713452s" totalImagesPullingTime="3.073979818s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:13:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:13:32.073607013 +0000 UTC m=+395.999533221" watchObservedRunningTime="2026-10-30 22:13:32.074713452 +0000 UTC m=+396.000639657"
Oct 30 22:13:32.109782 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:13:32.109709    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-x68fm" podStartSLOduration=2.004688205 podStartE2EDuration="5.109698735s" totalImagesPullingTime="3.10501053s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:13:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:13:32.108439633 +0000 UTC m=+396.034365839" watchObservedRunningTime="2026-10-30 22:13:32.109698735 +0000 UTC m=+396.035624941"
Oct 30 22:13:37.923968 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:13:37.923938    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-x69f9"
Oct 30 22:13:42.981749 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:13:42.981662    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-ssczl"
Oct 30 22:13:42.983618 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:13:42.983590    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-f29pm"
Oct 30 22:13:49.491980 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:13:49.491936    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6bz92"]
Oct 30 22:13:56.265788 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:13:56.265750    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 22:13:57.795606 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:13:57.795558    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5f875c48f7-tsrbc"]
Oct 30 22:13:57.971813 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:13:57.971776    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5f875c48f7-tsrbc"]
Oct 30 22:13:57.971953 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:13:57.971864    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f875c48f7-tsrbc"
Oct 30 22:13:57.985258 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:13:57.985239    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Oct 30 22:13:58.053453 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:13:58.053396    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/11204d2f-d63b-4ccc-a1fc-31e75acb6218-tls-cert\") pod \"authorino-5f875c48f7-tsrbc\" (UID: \"11204d2f-d63b-4ccc-a1fc-31e75acb6218\") " pod="kuadrant-system/authorino-5f875c48f7-tsrbc"
Oct 30 22:13:58.053558 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:13:58.053457    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7v4h\" (UniqueName: \"kubernetes.io/projected/11204d2f-d63b-4ccc-a1fc-31e75acb6218-kube-api-access-g7v4h\") pod \"authorino-5f875c48f7-tsrbc\" (UID: \"11204d2f-d63b-4ccc-a1fc-31e75acb6218\") " pod="kuadrant-system/authorino-5f875c48f7-tsrbc"
Oct 30 22:13:58.154458 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:13:58.154433    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/11204d2f-d63b-4ccc-a1fc-31e75acb6218-tls-cert\") pod \"authorino-5f875c48f7-tsrbc\" (UID: \"11204d2f-d63b-4ccc-a1fc-31e75acb6218\") " pod="kuadrant-system/authorino-5f875c48f7-tsrbc"
Oct 30 22:13:58.154562 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:13:58.154503    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7v4h\" (UniqueName: \"kubernetes.io/projected/11204d2f-d63b-4ccc-a1fc-31e75acb6218-kube-api-access-g7v4h\") pod \"authorino-5f875c48f7-tsrbc\" (UID: \"11204d2f-d63b-4ccc-a1fc-31e75acb6218\") " pod="kuadrant-system/authorino-5f875c48f7-tsrbc"
Oct 30 22:13:58.156693 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:13:58.156670    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/11204d2f-d63b-4ccc-a1fc-31e75acb6218-tls-cert\") pod \"authorino-5f875c48f7-tsrbc\" (UID: \"11204d2f-d63b-4ccc-a1fc-31e75acb6218\") " pod="kuadrant-system/authorino-5f875c48f7-tsrbc"
Oct 30 22:13:58.174844 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:13:58.174821    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7v4h\" (UniqueName: \"kubernetes.io/projected/11204d2f-d63b-4ccc-a1fc-31e75acb6218-kube-api-access-g7v4h\") pod \"authorino-5f875c48f7-tsrbc\" (UID: \"11204d2f-d63b-4ccc-a1fc-31e75acb6218\") " pod="kuadrant-system/authorino-5f875c48f7-tsrbc"
Oct 30 22:13:58.321849 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:13:58.321807    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f875c48f7-tsrbc"
Oct 30 22:13:58.647246 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:13:58.647215    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5f875c48f7-tsrbc"]
Oct 30 22:13:58.651116 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:13:58.651090    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod11204d2f_d63b_4ccc_a1fc_31e75acb6218.slice/crio-7504537134e6e7c61000c6bcf14087a3bf49ed7bfc766248596bbefc1534922f WatchSource:0}: Error finding container 7504537134e6e7c61000c6bcf14087a3bf49ed7bfc766248596bbefc1534922f: Status 404 returned error can't find the container with id 7504537134e6e7c61000c6bcf14087a3bf49ed7bfc766248596bbefc1534922f
Oct 30 22:13:59.098721 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:13:59.098577    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f875c48f7-tsrbc" event={"ID":"11204d2f-d63b-4ccc-a1fc-31e75acb6218","Type":"ContainerStarted","Data":"7504537134e6e7c61000c6bcf14087a3bf49ed7bfc766248596bbefc1534922f"}
Oct 30 22:14:00.104699 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:14:00.104618    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f875c48f7-tsrbc" event={"ID":"11204d2f-d63b-4ccc-a1fc-31e75acb6218","Type":"ContainerStarted","Data":"24fefc71b014e4b8bce636412edc806816a0d0125be012e9a35230632e119efc"}
Oct 30 22:14:00.139405 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:14:00.139362    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-5f875c48f7-tsrbc" podStartSLOduration=2.603524753 podStartE2EDuration="3.139349938s" totalImagesPullingTime="535.825185ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:13:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:14:00.138761652 +0000 UTC m=+424.064687857" watchObservedRunningTime="2026-10-30 22:14:00.139349938 +0000 UTC m=+424.065276144"
Oct 30 22:14:00.164549 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:14:00.164519    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-jx558"]
Oct 30 22:14:00.164705 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:14:00.164686    4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-jx558" podUID="afa19129-2247-41d0-bd7b-5cff3a9894d3" containerName="authorino" containerID="cri-o://feeeb196f08616fef9c57478d2802cf14f7d9d89ec69489e134c34d603e0d6ac" gracePeriod=30
Oct 30 22:14:00.413290 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:14:00.413264    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-jx558"
Oct 30 22:14:00.477940 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:14:00.477910    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afa19129-2247-41d0-bd7b-5cff3a9894d3-kube-api-access-znrm7\" (UniqueName: \"kubernetes.io/projected/afa19129-2247-41d0-bd7b-5cff3a9894d3-kube-api-access-znrm7\") pod \"afa19129-2247-41d0-bd7b-5cff3a9894d3\" (UID: \"afa19129-2247-41d0-bd7b-5cff3a9894d3\") "
Oct 30 22:14:00.479842 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:14:00.479818    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afa19129-2247-41d0-bd7b-5cff3a9894d3-kube-api-access-znrm7" pod "afa19129-2247-41d0-bd7b-5cff3a9894d3" (UID: "afa19129-2247-41d0-bd7b-5cff3a9894d3"). InnerVolumeSpecName "kube-api-access-znrm7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:00.579273 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:14:00.579249    4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znrm7\" (UniqueName: \"kubernetes.io/projected/afa19129-2247-41d0-bd7b-5cff3a9894d3-kube-api-access-znrm7\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:14:01.110007 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:14:01.109971    4477 generic.go:356] "Generic (PLEG): container finished" podID="afa19129-2247-41d0-bd7b-5cff3a9894d3" containerID="feeeb196f08616fef9c57478d2802cf14f7d9d89ec69489e134c34d603e0d6ac" exitCode=0
Oct 30 22:14:01.110489 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:14:01.110015    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-jx558"
Oct 30 22:14:01.110489 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:14:01.110072    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-jx558" event={"ID":"afa19129-2247-41d0-bd7b-5cff3a9894d3","Type":"ContainerDied","Data":"feeeb196f08616fef9c57478d2802cf14f7d9d89ec69489e134c34d603e0d6ac"}
Oct 30 22:14:01.110489 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:14:01.110110    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-jx558" event={"ID":"afa19129-2247-41d0-bd7b-5cff3a9894d3","Type":"ContainerDied","Data":"28565701ae8fabf74b18c20895dbebf1320d1a741779ff426b1f6a8d9dd8f207"}
Oct 30 22:14:01.110489 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:14:01.110128    4477 scope.go:122] "RemoveContainer" containerID="feeeb196f08616fef9c57478d2802cf14f7d9d89ec69489e134c34d603e0d6ac"
Oct 30 22:14:01.118411 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:14:01.118397    4477 scope.go:122] "RemoveContainer" containerID="feeeb196f08616fef9c57478d2802cf14f7d9d89ec69489e134c34d603e0d6ac"
Oct 30 22:14:01.118635 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:14:01.118614    4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"feeeb196f08616fef9c57478d2802cf14f7d9d89ec69489e134c34d603e0d6ac\": container with ID starting with feeeb196f08616fef9c57478d2802cf14f7d9d89ec69489e134c34d603e0d6ac not found: ID does not exist" containerID="feeeb196f08616fef9c57478d2802cf14f7d9d89ec69489e134c34d603e0d6ac"
Oct 30 22:14:01.118676 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:14:01.118643    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"feeeb196f08616fef9c57478d2802cf14f7d9d89ec69489e134c34d603e0d6ac"} err="failed to get container status \"feeeb196f08616fef9c57478d2802cf14f7d9d89ec69489e134c34d603e0d6ac\": rpc error: code = NotFound desc = could not find container \"feeeb196f08616fef9c57478d2802cf14f7d9d89ec69489e134c34d603e0d6ac\": container with ID starting with feeeb196f08616fef9c57478d2802cf14f7d9d89ec69489e134c34d603e0d6ac not found: ID does not exist"
Oct 30 22:14:01.156307 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:14:01.156285    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-jx558"]
Oct 30 22:14:01.166517 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:14:01.166498    4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-jx558"]
Oct 30 22:14:02.191557 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:14:02.191529    4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afa19129-2247-41d0-bd7b-5cff3a9894d3" path="/var/lib/kubelet/pods/afa19129-2247-41d0-bd7b-5cff3a9894d3/volumes"
Oct 30 22:14:02.980990 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:14:02.980958    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-x68fm"
Oct 30 22:14:02.983866 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:14:02.983846    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-ddzfb"
Oct 30 22:14:21.249237 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:14:21.249201    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-8lgfv"]
Oct 30 22:14:21.249717 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:14:21.249625    4477 cpu_manager.go:414] "containerMap: removing container" podUID="afa19129-2247-41d0-bd7b-5cff3a9894d3" containerName="authorino"
Oct 30 22:14:21.249717 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:14:21.249637    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afa19129-2247-41d0-bd7b-5cff3a9894d3" containerName="authorino"
Oct 30 22:14:21.249789 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:14:21.249750    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="afa19129-2247-41d0-bd7b-5cff3a9894d3" containerName="authorino"
Oct 30 22:14:21.251842 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:14:21.251823    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-8lgfv"
Oct 30 22:14:21.273148 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:14:21.273118    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-8lgfv"]
Oct 30 22:14:21.347394 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:14:21.347369    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8t4c4\" (UniqueName: \"kubernetes.io/projected/8bb0b4e6-3fa7-4257-b686-c6d84294038e-kube-api-access-8t4c4\") pod \"llmisvc-controller-manager-6466c554d8-8lgfv\" (UID: \"8bb0b4e6-3fa7-4257-b686-c6d84294038e\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-8lgfv"
Oct 30 22:14:21.347552 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:14:21.347404    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8bb0b4e6-3fa7-4257-b686-c6d84294038e-cert\") pod \"llmisvc-controller-manager-6466c554d8-8lgfv\" (UID: \"8bb0b4e6-3fa7-4257-b686-c6d84294038e\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-8lgfv"
Oct 30 22:14:21.448261 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:14:21.448233    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8t4c4\" (UniqueName: \"kubernetes.io/projected/8bb0b4e6-3fa7-4257-b686-c6d84294038e-kube-api-access-8t4c4\") pod \"llmisvc-controller-manager-6466c554d8-8lgfv\" (UID: \"8bb0b4e6-3fa7-4257-b686-c6d84294038e\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-8lgfv"
Oct 30 22:14:21.448370 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:14:21.448277    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8bb0b4e6-3fa7-4257-b686-c6d84294038e-cert\") pod \"llmisvc-controller-manager-6466c554d8-8lgfv\" (UID: \"8bb0b4e6-3fa7-4257-b686-c6d84294038e\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-8lgfv"
Oct 30 22:14:21.450653 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:14:21.450627    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8bb0b4e6-3fa7-4257-b686-c6d84294038e-cert\") pod \"llmisvc-controller-manager-6466c554d8-8lgfv\" (UID: \"8bb0b4e6-3fa7-4257-b686-c6d84294038e\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-8lgfv"
Oct 30 22:14:21.466549 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:14:21.466528    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8t4c4\" (UniqueName: \"kubernetes.io/projected/8bb0b4e6-3fa7-4257-b686-c6d84294038e-kube-api-access-8t4c4\") pod \"llmisvc-controller-manager-6466c554d8-8lgfv\" (UID: \"8bb0b4e6-3fa7-4257-b686-c6d84294038e\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-8lgfv"
Oct 30 22:14:21.563732 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:14:21.563681    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-8lgfv"
Oct 30 22:14:21.854218 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:14:21.854144    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-8lgfv"]
Oct 30 22:14:21.857863 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:14:21.857836    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bb0b4e6_3fa7_4257_b686_c6d84294038e.slice/crio-d443fd83b38345ae3b4c65d832a072d96283ae69ffdc273ee54465f4a56720cb WatchSource:0}: Error finding container d443fd83b38345ae3b4c65d832a072d96283ae69ffdc273ee54465f4a56720cb: Status 404 returned error can't find the container with id d443fd83b38345ae3b4c65d832a072d96283ae69ffdc273ee54465f4a56720cb
Oct 30 22:14:22.209526 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:14:22.209490    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-8lgfv" event={"ID":"8bb0b4e6-3fa7-4257-b686-c6d84294038e","Type":"ContainerStarted","Data":"da8fb06df7010931d3d820ad55bec1d5b01b1fc7ba097dbfc8e10c82fab079e1"}
Oct 30 22:14:22.209679 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:14:22.209548    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-8lgfv" event={"ID":"8bb0b4e6-3fa7-4257-b686-c6d84294038e","Type":"ContainerStarted","Data":"d443fd83b38345ae3b4c65d832a072d96283ae69ffdc273ee54465f4a56720cb"}
Oct 30 22:14:22.209745 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:14:22.209682    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-8lgfv"
Oct 30 22:14:22.247717 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:14:22.247675    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-8lgfv" podStartSLOduration=0.99688682 podStartE2EDuration="1.247663484s" totalImagesPullingTime="250.776664ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:14:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:14:22.247164654 +0000 UTC m=+446.173090860" watchObservedRunningTime="2026-10-30 22:14:22.247663484 +0000 UTC m=+446.173589730"
Oct 30 22:14:26.265998 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:14:26.265960    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 22:14:53.216665 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:14:53.216631    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-8lgfv"
Oct 30 22:14:53.279641 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:14:53.279608    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-x68fm"]
Oct 30 22:14:53.279834 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:14:53.279811    4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-x68fm" podUID="93a707d7-2921-438d-914b-193b1736f4f9" containerName="manager" containerID="cri-o://70a16fb6de6b57b7a80386de0412712cffc5e31245e9c84bae405a8a2ad25545" gracePeriod=30
Oct 30 22:14:53.525899 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:14:53.525875    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-x68fm"
Oct 30 22:14:53.614252 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:14:53.614223    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93a707d7-2921-438d-914b-193b1736f4f9-kube-api-access-pfq8d\" (UniqueName: \"kubernetes.io/projected/93a707d7-2921-438d-914b-193b1736f4f9-kube-api-access-pfq8d\") pod \"93a707d7-2921-438d-914b-193b1736f4f9\" (UID: \"93a707d7-2921-438d-914b-193b1736f4f9\") "
Oct 30 22:14:53.614382 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:14:53.614311    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93a707d7-2921-438d-914b-193b1736f4f9-cert\" (UniqueName: \"kubernetes.io/secret/93a707d7-2921-438d-914b-193b1736f4f9-cert\") pod \"93a707d7-2921-438d-914b-193b1736f4f9\" (UID: \"93a707d7-2921-438d-914b-193b1736f4f9\") "
Oct 30 22:14:53.616329 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:14:53.616295    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93a707d7-2921-438d-914b-193b1736f4f9-cert" pod "93a707d7-2921-438d-914b-193b1736f4f9" (UID: "93a707d7-2921-438d-914b-193b1736f4f9"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:14:53.616430 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:14:53.616371    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93a707d7-2921-438d-914b-193b1736f4f9-kube-api-access-pfq8d" pod "93a707d7-2921-438d-914b-193b1736f4f9" (UID: "93a707d7-2921-438d-914b-193b1736f4f9"). InnerVolumeSpecName "kube-api-access-pfq8d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:53.714863 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:14:53.714841    4477 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/93a707d7-2921-438d-914b-193b1736f4f9-cert\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:14:53.714863 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:14:53.714862    4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfq8d\" (UniqueName: \"kubernetes.io/projected/93a707d7-2921-438d-914b-193b1736f4f9-kube-api-access-pfq8d\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:14:54.353769 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:14:54.353736    4477 generic.go:356] "Generic (PLEG): container finished" podID="93a707d7-2921-438d-914b-193b1736f4f9" containerID="70a16fb6de6b57b7a80386de0412712cffc5e31245e9c84bae405a8a2ad25545" exitCode=0
Oct 30 22:14:54.354160 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:14:54.353785    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-x68fm" event={"ID":"93a707d7-2921-438d-914b-193b1736f4f9","Type":"ContainerDied","Data":"70a16fb6de6b57b7a80386de0412712cffc5e31245e9c84bae405a8a2ad25545"}
Oct 30 22:14:54.354160 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:14:54.353824    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-x68fm" event={"ID":"93a707d7-2921-438d-914b-193b1736f4f9","Type":"ContainerDied","Data":"51277fdd8e24365aee73e001c4095453add5419b288ad54e745a291ad91beb1f"}
Oct 30 22:14:54.354160 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:14:54.353845    4477 scope.go:122] "RemoveContainer" containerID="70a16fb6de6b57b7a80386de0412712cffc5e31245e9c84bae405a8a2ad25545"
Oct 30 22:14:54.354160 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:14:54.353787    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-x68fm"
Oct 30 22:14:54.362759 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:14:54.362738    4477 scope.go:122] "RemoveContainer" containerID="70a16fb6de6b57b7a80386de0412712cffc5e31245e9c84bae405a8a2ad25545"
Oct 30 22:14:54.363087 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:14:54.363065    4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70a16fb6de6b57b7a80386de0412712cffc5e31245e9c84bae405a8a2ad25545\": container with ID starting with 70a16fb6de6b57b7a80386de0412712cffc5e31245e9c84bae405a8a2ad25545 not found: ID does not exist" containerID="70a16fb6de6b57b7a80386de0412712cffc5e31245e9c84bae405a8a2ad25545"
Oct 30 22:14:54.363146 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:14:54.363096    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70a16fb6de6b57b7a80386de0412712cffc5e31245e9c84bae405a8a2ad25545"} err="failed to get container status \"70a16fb6de6b57b7a80386de0412712cffc5e31245e9c84bae405a8a2ad25545\": rpc error: code = NotFound desc = could not find container \"70a16fb6de6b57b7a80386de0412712cffc5e31245e9c84bae405a8a2ad25545\": container with ID starting with 70a16fb6de6b57b7a80386de0412712cffc5e31245e9c84bae405a8a2ad25545 not found: ID does not exist"
Oct 30 22:14:54.402441 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:14:54.402417    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-x68fm"]
Oct 30 22:14:54.413076 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:14:54.413055    4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-x68fm"]
Oct 30 22:14:56.198734 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:14:56.198695    4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93a707d7-2921-438d-914b-193b1736f4f9" path="/var/lib/kubelet/pods/93a707d7-2921-438d-914b-193b1736f4f9/volumes"
Oct 30 22:14:56.267085 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:14:56.267014    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 22:15:13.061200 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:15:13.061100    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6bz92"]
Oct 30 22:15:16.804534 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:15:16.804498    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6bz92"]
Oct 30 22:15:20.393950 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:15:20.393910    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6bz92"]
Oct 30 22:15:26.265289 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:15:26.265250    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 22:15:27.751383 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:15:27.751339    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6bz92"]
Oct 30 22:15:33.383580 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:15:33.383545    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6bz92"]
Oct 30 22:15:42.217488 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:15:42.217456    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5f875c48f7-tsrbc"]
Oct 30 22:15:42.218667 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:15:42.218617    4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-5f875c48f7-tsrbc" podUID="11204d2f-d63b-4ccc-a1fc-31e75acb6218" containerName="authorino" containerID="cri-o://24fefc71b014e4b8bce636412edc806816a0d0125be012e9a35230632e119efc" gracePeriod=30
Oct 30 22:15:42.478247 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:15:42.478194    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f875c48f7-tsrbc"
Oct 30 22:15:42.558299 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:15:42.558274    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11204d2f-d63b-4ccc-a1fc-31e75acb6218-kube-api-access-g7v4h\" (UniqueName: \"kubernetes.io/projected/11204d2f-d63b-4ccc-a1fc-31e75acb6218-kube-api-access-g7v4h\") pod \"11204d2f-d63b-4ccc-a1fc-31e75acb6218\" (UID: \"11204d2f-d63b-4ccc-a1fc-31e75acb6218\") "
Oct 30 22:15:42.558425 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:15:42.558362    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11204d2f-d63b-4ccc-a1fc-31e75acb6218-tls-cert\" (UniqueName: \"kubernetes.io/secret/11204d2f-d63b-4ccc-a1fc-31e75acb6218-tls-cert\") pod \"11204d2f-d63b-4ccc-a1fc-31e75acb6218\" (UID: \"11204d2f-d63b-4ccc-a1fc-31e75acb6218\") "
Oct 30 22:15:42.560441 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:15:42.560410    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11204d2f-d63b-4ccc-a1fc-31e75acb6218-kube-api-access-g7v4h" pod "11204d2f-d63b-4ccc-a1fc-31e75acb6218" (UID: "11204d2f-d63b-4ccc-a1fc-31e75acb6218"). InnerVolumeSpecName "kube-api-access-g7v4h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:42.570404 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:15:42.570379    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11204d2f-d63b-4ccc-a1fc-31e75acb6218-tls-cert" pod "11204d2f-d63b-4ccc-a1fc-31e75acb6218" (UID: "11204d2f-d63b-4ccc-a1fc-31e75acb6218"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:15:42.584630 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:15:42.584605    4477 generic.go:356] "Generic (PLEG): container finished" podID="11204d2f-d63b-4ccc-a1fc-31e75acb6218" containerID="24fefc71b014e4b8bce636412edc806816a0d0125be012e9a35230632e119efc" exitCode=0
Oct 30 22:15:42.584710 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:15:42.584647    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f875c48f7-tsrbc"
Oct 30 22:15:42.584710 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:15:42.584680    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f875c48f7-tsrbc" event={"ID":"11204d2f-d63b-4ccc-a1fc-31e75acb6218","Type":"ContainerDied","Data":"24fefc71b014e4b8bce636412edc806816a0d0125be012e9a35230632e119efc"}
Oct 30 22:15:42.584781 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:15:42.584721    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f875c48f7-tsrbc" event={"ID":"11204d2f-d63b-4ccc-a1fc-31e75acb6218","Type":"ContainerDied","Data":"7504537134e6e7c61000c6bcf14087a3bf49ed7bfc766248596bbefc1534922f"}
Oct 30 22:15:42.584781 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:15:42.584737    4477 scope.go:122] "RemoveContainer" containerID="24fefc71b014e4b8bce636412edc806816a0d0125be012e9a35230632e119efc"
Oct 30 22:15:42.593190 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:15:42.593175    4477 scope.go:122] "RemoveContainer" containerID="24fefc71b014e4b8bce636412edc806816a0d0125be012e9a35230632e119efc"
Oct 30 22:15:42.593432 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:15:42.593408    4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24fefc71b014e4b8bce636412edc806816a0d0125be012e9a35230632e119efc\": container with ID starting with 24fefc71b014e4b8bce636412edc806816a0d0125be012e9a35230632e119efc not found: ID does not exist" containerID="24fefc71b014e4b8bce636412edc806816a0d0125be012e9a35230632e119efc"
Oct 30 22:15:42.593495 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:15:42.593440    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24fefc71b014e4b8bce636412edc806816a0d0125be012e9a35230632e119efc"} err="failed to get container status \"24fefc71b014e4b8bce636412edc806816a0d0125be012e9a35230632e119efc\": rpc error: code = NotFound desc = could not find container \"24fefc71b014e4b8bce636412edc806816a0d0125be012e9a35230632e119efc\": container with ID starting with 24fefc71b014e4b8bce636412edc806816a0d0125be012e9a35230632e119efc not found: ID does not exist"
Oct 30 22:15:42.630112 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:15:42.630091    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5f875c48f7-tsrbc"]
Oct 30 22:15:42.633893 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:15:42.633865    4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5f875c48f7-tsrbc"]
Oct 30 22:15:42.659021 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:15:42.658995    4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7v4h\" (UniqueName: \"kubernetes.io/projected/11204d2f-d63b-4ccc-a1fc-31e75acb6218-kube-api-access-g7v4h\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:15:42.659021 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:15:42.659018    4477 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/11204d2f-d63b-4ccc-a1fc-31e75acb6218-tls-cert\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:15:44.193176 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:15:44.193139    4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11204d2f-d63b-4ccc-a1fc-31e75acb6218" path="/var/lib/kubelet/pods/11204d2f-d63b-4ccc-a1fc-31e75acb6218/volumes"
Oct 30 22:15:50.878436 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:15:50.878393    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-frdpl"]
Oct 30 22:15:50.879031 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:15:50.878987    4477 cpu_manager.go:414] "containerMap: removing container" podUID="11204d2f-d63b-4ccc-a1fc-31e75acb6218" containerName="authorino"
Oct 30 22:15:50.879031 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:15:50.879013    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11204d2f-d63b-4ccc-a1fc-31e75acb6218" containerName="authorino"
Oct 30 22:15:50.879031 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:15:50.879031    4477 cpu_manager.go:414] "containerMap: removing container" podUID="93a707d7-2921-438d-914b-193b1736f4f9" containerName="manager"
Oct 30 22:15:50.879484 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:15:50.879036    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93a707d7-2921-438d-914b-193b1736f4f9" containerName="manager"
Oct 30 22:15:50.879484 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:15:50.879190    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="11204d2f-d63b-4ccc-a1fc-31e75acb6218" containerName="authorino"
Oct 30 22:15:50.879484 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:15:50.879204    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="93a707d7-2921-438d-914b-193b1736f4f9" containerName="manager"
Oct 30 22:15:50.883732 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:15:50.883714    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-frdpl"
Oct 30 22:15:50.896784 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:15:50.896766    4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 30 22:15:50.909634 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:15:50.909590    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-frdpl"]
Oct 30 22:15:50.923302 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:15:50.923279    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3b4a8a3-deec-47d0-9692-4f44a319f922-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-frdpl\" (UID: \"d3b4a8a3-deec-47d0-9692-4f44a319f922\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-frdpl"
Oct 30 22:15:50.923420 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:15:50.923312    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3b4a8a3-deec-47d0-9692-4f44a319f922-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-frdpl\" (UID: \"d3b4a8a3-deec-47d0-9692-4f44a319f922\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-frdpl"
Oct 30 22:15:50.923625 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:15:50.923602    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3b4a8a3-deec-47d0-9692-4f44a319f922-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-frdpl\" (UID: \"d3b4a8a3-deec-47d0-9692-4f44a319f922\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-frdpl"
Oct 30 22:15:50.923695 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:15:50.923677    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87jb5\" (UniqueName: \"kubernetes.io/projected/d3b4a8a3-deec-47d0-9692-4f44a319f922-kube-api-access-87jb5\") pod \"maas-default-gateway-openshift-default-6db586dd6c-frdpl\" (UID: \"d3b4a8a3-deec-47d0-9692-4f44a319f922\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-frdpl"
Oct 30 22:15:50.923741 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:15:50.923717    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3b4a8a3-deec-47d0-9692-4f44a319f922-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-frdpl\" (UID: \"d3b4a8a3-deec-47d0-9692-4f44a319f922\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-frdpl"
Oct 30 22:15:50.923792 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:15:50.923746    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3b4a8a3-deec-47d0-9692-4f44a319f922-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-frdpl\" (UID: \"d3b4a8a3-deec-47d0-9692-4f44a319f922\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-frdpl"
Oct 30 22:15:50.923792 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:15:50.923774    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3b4a8a3-deec-47d0-9692-4f44a319f922-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-frdpl\" (UID: \"d3b4a8a3-deec-47d0-9692-4f44a319f922\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-frdpl"
Oct 30 22:15:50.923901 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:15:50.923807    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3b4a8a3-deec-47d0-9692-4f44a319f922-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-frdpl\" (UID: \"d3b4a8a3-deec-47d0-9692-4f44a319f922\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-frdpl"
Oct 30 22:15:50.923901 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:15:50.923874    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3b4a8a3-deec-47d0-9692-4f44a319f922-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-frdpl\" (UID: \"d3b4a8a3-deec-47d0-9692-4f44a319f922\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-frdpl"
Oct 30 22:15:50.986843 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:15:50.986817    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-j98cq\""
Oct 30 22:15:51.025129 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:15:51.025103    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87jb5\" (UniqueName: \"kubernetes.io/projected/d3b4a8a3-deec-47d0-9692-4f44a319f922-kube-api-access-87jb5\") pod \"maas-default-gateway-openshift-default-6db586dd6c-frdpl\" (UID: \"d3b4a8a3-deec-47d0-9692-4f44a319f922\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-frdpl"
Oct 30 22:15:51.025247 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:15:51.025143    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3b4a8a3-deec-47d0-9692-4f44a319f922-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-frdpl\" (UID: \"d3b4a8a3-deec-47d0-9692-4f44a319f922\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-frdpl"
Oct 30 22:15:51.025314 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:15:51.025263    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3b4a8a3-deec-47d0-9692-4f44a319f922-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-frdpl\" (UID: \"d3b4a8a3-deec-47d0-9692-4f44a319f922\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-frdpl"
Oct 30 22:15:51.025314 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:15:51.025297    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3b4a8a3-deec-47d0-9692-4f44a319f922-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-frdpl\" (UID: \"d3b4a8a3-deec-47d0-9692-4f44a319f922\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-frdpl"
Oct 30 22:15:51.025442 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:15:51.025420    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3b4a8a3-deec-47d0-9692-4f44a319f922-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-frdpl\" (UID: \"d3b4a8a3-deec-47d0-9692-4f44a319f922\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-frdpl"
Oct 30 22:15:51.025506 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:15:51.025458    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3b4a8a3-deec-47d0-9692-4f44a319f922-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-frdpl\" (UID: \"d3b4a8a3-deec-47d0-9692-4f44a319f922\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-frdpl"
Oct 30 22:15:51.025562 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:15:51.025516    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3b4a8a3-deec-47d0-9692-4f44a319f922-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-frdpl\" (UID: \"d3b4a8a3-deec-47d0-9692-4f44a319f922\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-frdpl"
Oct 30 22:15:51.025562 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:15:51.025529    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3b4a8a3-deec-47d0-9692-4f44a319f922-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-frdpl\" (UID: \"d3b4a8a3-deec-47d0-9692-4f44a319f922\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-frdpl"
Oct 30 22:15:51.025562 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:15:51.025541    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3b4a8a3-deec-47d0-9692-4f44a319f922-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-frdpl\" (UID: \"d3b4a8a3-deec-47d0-9692-4f44a319f922\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-frdpl"
Oct 30 22:15:51.025709 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:15:51.025570    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3b4a8a3-deec-47d0-9692-4f44a319f922-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-frdpl\" (UID: \"d3b4a8a3-deec-47d0-9692-4f44a319f922\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-frdpl"
Oct 30 22:15:51.025709 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:15:51.025664    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3b4a8a3-deec-47d0-9692-4f44a319f922-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-frdpl\" (UID: \"d3b4a8a3-deec-47d0-9692-4f44a319f922\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-frdpl"
Oct 30 22:15:51.025815 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:15:51.025715    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3b4a8a3-deec-47d0-9692-4f44a319f922-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-frdpl\" (UID: \"d3b4a8a3-deec-47d0-9692-4f44a319f922\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-frdpl"
Oct 30 22:15:51.025971 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:15:51.025941    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3b4a8a3-deec-47d0-9692-4f44a319f922-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-frdpl\" (UID: \"d3b4a8a3-deec-47d0-9692-4f44a319f922\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-frdpl"
Oct 30 22:15:51.026026 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:15:51.025991    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3b4a8a3-deec-47d0-9692-4f44a319f922-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-frdpl\" (UID: \"d3b4a8a3-deec-47d0-9692-4f44a319f922\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-frdpl"
Oct 30 22:15:51.027449 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:15:51.027429    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3b4a8a3-deec-47d0-9692-4f44a319f922-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-frdpl\" (UID: \"d3b4a8a3-deec-47d0-9692-4f44a319f922\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-frdpl"
Oct 30 22:15:51.027759 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:15:51.027740    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3b4a8a3-deec-47d0-9692-4f44a319f922-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-frdpl\" (UID: \"d3b4a8a3-deec-47d0-9692-4f44a319f922\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-frdpl"
Oct 30 22:15:51.042508 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:15:51.042487    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87jb5\" (UniqueName: \"kubernetes.io/projected/d3b4a8a3-deec-47d0-9692-4f44a319f922-kube-api-access-87jb5\") pod \"maas-default-gateway-openshift-default-6db586dd6c-frdpl\" (UID: \"d3b4a8a3-deec-47d0-9692-4f44a319f922\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-frdpl"
Oct 30 22:15:51.042591 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:15:51.042549    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3b4a8a3-deec-47d0-9692-4f44a319f922-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-frdpl\" (UID: \"d3b4a8a3-deec-47d0-9692-4f44a319f922\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-frdpl"
Oct 30 22:15:51.197655 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:15:51.197598    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-frdpl"
Oct 30 22:15:51.507465 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:15:51.507364    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-frdpl"]
Oct 30 22:15:51.512619 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:15:51.512594    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3b4a8a3_deec_47d0_9692_4f44a319f922.slice/crio-8415c50c98f4464565b26d369188e21a1149f86558b34d402cb007f4cbc7d941 WatchSource:0}: Error finding container 8415c50c98f4464565b26d369188e21a1149f86558b34d402cb007f4cbc7d941: Status 404 returned error can't find the container with id 8415c50c98f4464565b26d369188e21a1149f86558b34d402cb007f4cbc7d941
Oct 30 22:15:51.639087 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:15:51.639055    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-frdpl" event={"ID":"d3b4a8a3-deec-47d0-9692-4f44a319f922","Type":"ContainerStarted","Data":"8415c50c98f4464565b26d369188e21a1149f86558b34d402cb007f4cbc7d941"}
Oct 30 22:15:54.540604 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:15:54.540567    4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:15:54.541007 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:15:54.540651    4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:15:54.541007 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:15:54.540696    4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:15:54.657385 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:15:54.657353    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-frdpl" event={"ID":"d3b4a8a3-deec-47d0-9692-4f44a319f922","Type":"ContainerStarted","Data":"67263e866672e1e481a94bc3b912551703ef7626d36390da30239e3c097f1de2"}
Oct 30 22:15:54.705957 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:15:54.705905    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-frdpl" podStartSLOduration=1.680707565 podStartE2EDuration="4.705892204s" totalImagesPullingTime="3.025184639s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:54.704822568 +0000 UTC m=+538.630748794" watchObservedRunningTime="2026-10-30 22:15:54.705892204 +0000 UTC m=+538.631818410"
Oct 30 22:15:55.198468 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:15:55.198432    4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-frdpl"
Oct 30 22:15:55.199954 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:15:55.199924    4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-frdpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Oct 30 22:15:55.200084 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:15:55.199988    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-frdpl" podUID="d3b4a8a3-deec-47d0-9692-4f44a319f922" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Oct 30 22:15:56.198424 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:15:56.198395    4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-frdpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Oct 30 22:15:56.198771 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:15:56.198451    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-frdpl" podUID="d3b4a8a3-deec-47d0-9692-4f44a319f922" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Oct 30 22:15:56.265251 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:15:56.265219    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 22:15:57.198530 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:15:57.198495    4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-frdpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Oct 30 22:15:57.198928 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:15:57.198558    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-frdpl" podUID="d3b4a8a3-deec-47d0-9692-4f44a319f922" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Oct 30 22:15:58.198661 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:15:58.198631    4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-frdpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Oct 30 22:15:58.199157 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:15:58.198695    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-frdpl" podUID="d3b4a8a3-deec-47d0-9692-4f44a319f922" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Oct 30 22:15:59.198385 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:15:59.198351    4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-frdpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Oct 30 22:15:59.198560 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:15:59.198401    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-frdpl" podUID="d3b4a8a3-deec-47d0-9692-4f44a319f922" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Oct 30 22:16:00.198016 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:00.197985    4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-frdpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Oct 30 22:16:00.198487 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:00.198056    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-frdpl" podUID="d3b4a8a3-deec-47d0-9692-4f44a319f922" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Oct 30 22:16:01.198745 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:01.198706    4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-frdpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Oct 30 22:16:01.199137 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:01.198780    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-frdpl" podUID="d3b4a8a3-deec-47d0-9692-4f44a319f922" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Oct 30 22:16:02.665454 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:02.665396    4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-frdpl"
Oct 30 22:16:02.666192 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:02.666035    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-frdpl"
Oct 30 22:16:02.667056 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:02.667014    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-frdpl"
Oct 30 22:16:05.913485 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:05.913455    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-bjfxc"]
Oct 30 22:16:05.918775 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:05.918757    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-bjfxc"
Oct 30 22:16:05.944267 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:05.944241    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-bjfxc"]
Oct 30 22:16:05.984578 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:05.984550    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzz52\" (UniqueName: \"kubernetes.io/projected/eeb7af55-f99f-411e-9f71-8ac1638c67ed-kube-api-access-nzz52\") pod \"maas-default-gateway-openshift-default-6db586dd6c-bjfxc\" (UID: \"eeb7af55-f99f-411e-9f71-8ac1638c67ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-bjfxc"
Oct 30 22:16:05.984690 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:05.984581    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eeb7af55-f99f-411e-9f71-8ac1638c67ed-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-bjfxc\" (UID: \"eeb7af55-f99f-411e-9f71-8ac1638c67ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-bjfxc"
Oct 30 22:16:05.984690 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:05.984617    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eeb7af55-f99f-411e-9f71-8ac1638c67ed-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-bjfxc\" (UID: \"eeb7af55-f99f-411e-9f71-8ac1638c67ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-bjfxc"
Oct 30 22:16:05.984769 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:05.984708    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeb7af55-f99f-411e-9f71-8ac1638c67ed-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-bjfxc\" (UID: \"eeb7af55-f99f-411e-9f71-8ac1638c67ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-bjfxc"
Oct 30 22:16:05.984769 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:05.984753    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eeb7af55-f99f-411e-9f71-8ac1638c67ed-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-bjfxc\" (UID: \"eeb7af55-f99f-411e-9f71-8ac1638c67ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-bjfxc"
Oct 30 22:16:05.984844 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:05.984786    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eeb7af55-f99f-411e-9f71-8ac1638c67ed-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-bjfxc\" (UID: \"eeb7af55-f99f-411e-9f71-8ac1638c67ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-bjfxc"
Oct 30 22:16:05.984911 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:05.984889    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeb7af55-f99f-411e-9f71-8ac1638c67ed-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-bjfxc\" (UID: \"eeb7af55-f99f-411e-9f71-8ac1638c67ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-bjfxc"
Oct 30 22:16:05.984945 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:05.984923    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eeb7af55-f99f-411e-9f71-8ac1638c67ed-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-bjfxc\" (UID: \"eeb7af55-f99f-411e-9f71-8ac1638c67ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-bjfxc"
Oct 30 22:16:05.984996 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:05.984947    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eeb7af55-f99f-411e-9f71-8ac1638c67ed-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-bjfxc\" (UID: \"eeb7af55-f99f-411e-9f71-8ac1638c67ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-bjfxc"
Oct 30 22:16:06.085965 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:06.085940    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eeb7af55-f99f-411e-9f71-8ac1638c67ed-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-bjfxc\" (UID: \"eeb7af55-f99f-411e-9f71-8ac1638c67ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-bjfxc"
Oct 30 22:16:06.086081 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:06.085980    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eeb7af55-f99f-411e-9f71-8ac1638c67ed-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-bjfxc\" (UID: \"eeb7af55-f99f-411e-9f71-8ac1638c67ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-bjfxc"
Oct 30 22:16:06.086081 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:06.086002    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeb7af55-f99f-411e-9f71-8ac1638c67ed-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-bjfxc\" (UID: \"eeb7af55-f99f-411e-9f71-8ac1638c67ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-bjfxc"
Oct 30 22:16:06.086081 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:06.086028    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eeb7af55-f99f-411e-9f71-8ac1638c67ed-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-bjfxc\" (UID: \"eeb7af55-f99f-411e-9f71-8ac1638c67ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-bjfxc"
Oct 30 22:16:06.086222 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:06.086093    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eeb7af55-f99f-411e-9f71-8ac1638c67ed-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-bjfxc\" (UID: \"eeb7af55-f99f-411e-9f71-8ac1638c67ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-bjfxc"
Oct 30 22:16:06.086222 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:06.086164    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeb7af55-f99f-411e-9f71-8ac1638c67ed-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-bjfxc\" (UID: \"eeb7af55-f99f-411e-9f71-8ac1638c67ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-bjfxc"
Oct 30 22:16:06.086222 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:06.086192    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eeb7af55-f99f-411e-9f71-8ac1638c67ed-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-bjfxc\" (UID: \"eeb7af55-f99f-411e-9f71-8ac1638c67ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-bjfxc"
Oct 30 22:16:06.086365 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:06.086221    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eeb7af55-f99f-411e-9f71-8ac1638c67ed-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-bjfxc\" (UID: \"eeb7af55-f99f-411e-9f71-8ac1638c67ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-bjfxc"
Oct 30 22:16:06.086518 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:06.086432    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzz52\" (UniqueName: \"kubernetes.io/projected/eeb7af55-f99f-411e-9f71-8ac1638c67ed-kube-api-access-nzz52\") pod \"maas-default-gateway-openshift-default-6db586dd6c-bjfxc\" (UID: \"eeb7af55-f99f-411e-9f71-8ac1638c67ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-bjfxc"
Oct 30 22:16:06.086518 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:06.086495    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeb7af55-f99f-411e-9f71-8ac1638c67ed-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-bjfxc\" (UID: \"eeb7af55-f99f-411e-9f71-8ac1638c67ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-bjfxc"
Oct 30 22:16:06.086518 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:06.086512    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eeb7af55-f99f-411e-9f71-8ac1638c67ed-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-bjfxc\" (UID: \"eeb7af55-f99f-411e-9f71-8ac1638c67ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-bjfxc"
Oct 30 22:16:06.086695 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:06.086655    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eeb7af55-f99f-411e-9f71-8ac1638c67ed-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-bjfxc\" (UID: \"eeb7af55-f99f-411e-9f71-8ac1638c67ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-bjfxc"
Oct 30 22:16:06.086757 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:06.086738    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eeb7af55-f99f-411e-9f71-8ac1638c67ed-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-bjfxc\" (UID: \"eeb7af55-f99f-411e-9f71-8ac1638c67ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-bjfxc"
Oct 30 22:16:06.086913 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:06.086890    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eeb7af55-f99f-411e-9f71-8ac1638c67ed-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-bjfxc\" (UID: \"eeb7af55-f99f-411e-9f71-8ac1638c67ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-bjfxc"
Oct 30 22:16:06.088261 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:06.088241    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eeb7af55-f99f-411e-9f71-8ac1638c67ed-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-bjfxc\" (UID: \"eeb7af55-f99f-411e-9f71-8ac1638c67ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-bjfxc"
Oct 30 22:16:06.088479 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:06.088462    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eeb7af55-f99f-411e-9f71-8ac1638c67ed-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-bjfxc\" (UID: \"eeb7af55-f99f-411e-9f71-8ac1638c67ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-bjfxc"
Oct 30 22:16:06.107747 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:06.107721    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eeb7af55-f99f-411e-9f71-8ac1638c67ed-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-bjfxc\" (UID: \"eeb7af55-f99f-411e-9f71-8ac1638c67ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-bjfxc"
Oct 30 22:16:06.107929 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:06.107912    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzz52\" (UniqueName: \"kubernetes.io/projected/eeb7af55-f99f-411e-9f71-8ac1638c67ed-kube-api-access-nzz52\") pod \"maas-default-gateway-openshift-default-6db586dd6c-bjfxc\" (UID: \"eeb7af55-f99f-411e-9f71-8ac1638c67ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-bjfxc"
Oct 30 22:16:06.231957 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:06.231900    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-bjfxc"
Oct 30 22:16:06.536510 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:06.536442    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-bjfxc"]
Oct 30 22:16:06.542798 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:16:06.542772    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeeb7af55_f99f_411e_9f71_8ac1638c67ed.slice/crio-5e470633a6fe2bb362f28acd68c2af2cb2be8afdb73a85b3f25f92ec83bcc564 WatchSource:0}: Error finding container 5e470633a6fe2bb362f28acd68c2af2cb2be8afdb73a85b3f25f92ec83bcc564: Status 404 returned error can't find the container with id 5e470633a6fe2bb362f28acd68c2af2cb2be8afdb73a85b3f25f92ec83bcc564
Oct 30 22:16:06.544727 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:06.544688    4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:16:06.544914 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:06.544887    4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:16:06.544989 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:06.544934    4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:16:06.734445 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:06.734411    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-bjfxc" event={"ID":"eeb7af55-f99f-411e-9f71-8ac1638c67ed","Type":"ContainerStarted","Data":"b7087d3485f3e07be6fd62c2225b5ebb7a99e76c2c45328e1917ec8d91061f84"}
Oct 30 22:16:06.734591 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:06.734460    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-bjfxc" event={"ID":"eeb7af55-f99f-411e-9f71-8ac1638c67ed","Type":"ContainerStarted","Data":"5e470633a6fe2bb362f28acd68c2af2cb2be8afdb73a85b3f25f92ec83bcc564"}
Oct 30 22:16:06.773763 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:06.773709    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-bjfxc" podStartSLOduration=1.773692245 podStartE2EDuration="1.773692245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:16:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:16:06.772717082 +0000 UTC m=+550.698643453" watchObservedRunningTime="2026-10-30 22:16:06.773692245 +0000 UTC m=+550.699618453"
Oct 30 22:16:07.232600 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:07.232566    4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-bjfxc"
Oct 30 22:16:07.234103 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:07.234075    4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-bjfxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Oct 30 22:16:07.234233 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:07.234124    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-bjfxc" podUID="eeb7af55-f99f-411e-9f71-8ac1638c67ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Oct 30 22:16:08.232477 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:08.232443    4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-bjfxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Oct 30 22:16:08.232658 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:08.232512    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-bjfxc" podUID="eeb7af55-f99f-411e-9f71-8ac1638c67ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Oct 30 22:16:09.233246 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:09.233209    4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-bjfxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Oct 30 22:16:09.233722 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:09.233280    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-bjfxc" podUID="eeb7af55-f99f-411e-9f71-8ac1638c67ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Oct 30 22:16:10.233108 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:10.233077    4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-bjfxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Oct 30 22:16:10.233281 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:10.233124    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-bjfxc" podUID="eeb7af55-f99f-411e-9f71-8ac1638c67ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Oct 30 22:16:11.232395 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:11.232359    4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-bjfxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Oct 30 22:16:11.232576 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:11.232414    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-bjfxc" podUID="eeb7af55-f99f-411e-9f71-8ac1638c67ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Oct 30 22:16:12.232303 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:12.232266    4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-bjfxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Oct 30 22:16:12.232694 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:12.232325    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-bjfxc" podUID="eeb7af55-f99f-411e-9f71-8ac1638c67ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Oct 30 22:16:13.232707 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:13.232668    4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-bjfxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Oct 30 22:16:13.233168 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:13.232742    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-bjfxc" podUID="eeb7af55-f99f-411e-9f71-8ac1638c67ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Oct 30 22:16:14.694785 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:14.694733    4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-bjfxc"
Oct 30 22:16:14.695739 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:14.695721    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-bjfxc"
Oct 30 22:16:14.696285 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:14.696268    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-bjfxc"
Oct 30 22:16:26.265731 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:16:26.265628    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 22:16:49.543491 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:49.543411    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6bz92"]
Oct 30 22:16:49.548799 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:49.548776    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-2b1e88-6b57666d9d-7lhjc"]
Oct 30 22:16:49.552476 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:49.552460    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-2b1e88-6b57666d9d-7lhjc"
Oct 30 22:16:49.564114 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:49.564090    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c67bfb3e-cc4c-40a2-9d8f-bc36f48f2a2f-metrics-certs\") pod \"maas-api-e2e-worker-w1-2b1e88-6b57666d9d-7lhjc\" (UID: \"c67bfb3e-cc4c-40a2-9d8f-bc36f48f2a2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-2b1e88-6b57666d9d-7lhjc"
Oct 30 22:16:49.564213 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:49.564142    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c67bfb3e-cc4c-40a2-9d8f-bc36f48f2a2f-maas-api-tls\") pod \"maas-api-e2e-worker-w1-2b1e88-6b57666d9d-7lhjc\" (UID: \"c67bfb3e-cc4c-40a2-9d8f-bc36f48f2a2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-2b1e88-6b57666d9d-7lhjc"
Oct 30 22:16:49.564213 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:49.564181    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c67bfb3e-cc4c-40a2-9d8f-bc36f48f2a2f-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-2b1e88-6b57666d9d-7lhjc\" (UID: \"c67bfb3e-cc4c-40a2-9d8f-bc36f48f2a2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-2b1e88-6b57666d9d-7lhjc"
Oct 30 22:16:49.564342 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:49.564295    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmnth\" (UniqueName: \"kubernetes.io/projected/c67bfb3e-cc4c-40a2-9d8f-bc36f48f2a2f-kube-api-access-dmnth\") pod \"maas-api-e2e-worker-w1-2b1e88-6b57666d9d-7lhjc\" (UID: \"c67bfb3e-cc4c-40a2-9d8f-bc36f48f2a2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-2b1e88-6b57666d9d-7lhjc"
Oct 30 22:16:49.569909 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:49.569889    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-2b1e88-serving-cert\""
Oct 30 22:16:49.570021 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:49.570010    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-l95g7\""
Oct 30 22:16:49.570102 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:49.570067    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 22:16:49.582532 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:49.582509    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-2b1e88-6b57666d9d-7lhjc"]
Oct 30 22:16:49.664947 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:49.664912    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c67bfb3e-cc4c-40a2-9d8f-bc36f48f2a2f-metrics-certs\") pod \"maas-api-e2e-worker-w1-2b1e88-6b57666d9d-7lhjc\" (UID: \"c67bfb3e-cc4c-40a2-9d8f-bc36f48f2a2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-2b1e88-6b57666d9d-7lhjc"
Oct 30 22:16:49.665104 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:49.664960    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c67bfb3e-cc4c-40a2-9d8f-bc36f48f2a2f-maas-api-tls\") pod \"maas-api-e2e-worker-w1-2b1e88-6b57666d9d-7lhjc\" (UID: \"c67bfb3e-cc4c-40a2-9d8f-bc36f48f2a2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-2b1e88-6b57666d9d-7lhjc"
Oct 30 22:16:49.665104 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:49.664985    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c67bfb3e-cc4c-40a2-9d8f-bc36f48f2a2f-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-2b1e88-6b57666d9d-7lhjc\" (UID: \"c67bfb3e-cc4c-40a2-9d8f-bc36f48f2a2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-2b1e88-6b57666d9d-7lhjc"
Oct 30 22:16:49.665104 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:49.665005    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmnth\" (UniqueName: \"kubernetes.io/projected/c67bfb3e-cc4c-40a2-9d8f-bc36f48f2a2f-kube-api-access-dmnth\") pod \"maas-api-e2e-worker-w1-2b1e88-6b57666d9d-7lhjc\" (UID: \"c67bfb3e-cc4c-40a2-9d8f-bc36f48f2a2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-2b1e88-6b57666d9d-7lhjc"
Oct 30 22:16:49.665104 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:16:49.665080    4477 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w1-2b1e88-serving-cert: secret "maas-api-e2e-worker-w1-2b1e88-serving-cert" not found
Oct 30 22:16:49.665277 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:16:49.665143    4477 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c67bfb3e-cc4c-40a2-9d8f-bc36f48f2a2f-maas-api-tls podName:c67bfb3e-cc4c-40a2-9d8f-bc36f48f2a2f nodeName:}" failed. No retries permitted until 2026-10-30 22:16:50.165128568 +0000 UTC m=+594.091054775 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c67bfb3e-cc4c-40a2-9d8f-bc36f48f2a2f-maas-api-tls") pod "maas-api-e2e-worker-w1-2b1e88-6b57666d9d-7lhjc" (UID: "c67bfb3e-cc4c-40a2-9d8f-bc36f48f2a2f") : secret "maas-api-e2e-worker-w1-2b1e88-serving-cert" not found
Oct 30 22:16:49.665709 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:49.665692    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c67bfb3e-cc4c-40a2-9d8f-bc36f48f2a2f-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-2b1e88-6b57666d9d-7lhjc\" (UID: \"c67bfb3e-cc4c-40a2-9d8f-bc36f48f2a2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-2b1e88-6b57666d9d-7lhjc"
Oct 30 22:16:49.667300 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:49.667285    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c67bfb3e-cc4c-40a2-9d8f-bc36f48f2a2f-metrics-certs\") pod \"maas-api-e2e-worker-w1-2b1e88-6b57666d9d-7lhjc\" (UID: \"c67bfb3e-cc4c-40a2-9d8f-bc36f48f2a2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-2b1e88-6b57666d9d-7lhjc"
Oct 30 22:16:49.685511 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:49.685483    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmnth\" (UniqueName: \"kubernetes.io/projected/c67bfb3e-cc4c-40a2-9d8f-bc36f48f2a2f-kube-api-access-dmnth\") pod \"maas-api-e2e-worker-w1-2b1e88-6b57666d9d-7lhjc\" (UID: \"c67bfb3e-cc4c-40a2-9d8f-bc36f48f2a2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-2b1e88-6b57666d9d-7lhjc"
Oct 30 22:16:50.169840 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:50.169809    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c67bfb3e-cc4c-40a2-9d8f-bc36f48f2a2f-maas-api-tls\") pod \"maas-api-e2e-worker-w1-2b1e88-6b57666d9d-7lhjc\" (UID: \"c67bfb3e-cc4c-40a2-9d8f-bc36f48f2a2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-2b1e88-6b57666d9d-7lhjc"
Oct 30 22:16:50.172070 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:50.172030    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c67bfb3e-cc4c-40a2-9d8f-bc36f48f2a2f-maas-api-tls\") pod \"maas-api-e2e-worker-w1-2b1e88-6b57666d9d-7lhjc\" (UID: \"c67bfb3e-cc4c-40a2-9d8f-bc36f48f2a2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-2b1e88-6b57666d9d-7lhjc"
Oct 30 22:16:50.465941 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:50.465856    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-2b1e88-6b57666d9d-7lhjc"
Oct 30 22:16:50.764926 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:50.764765    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-2b1e88-6b57666d9d-7lhjc"]
Oct 30 22:16:50.769649 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:16:50.769622    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc67bfb3e_cc4c_40a2_9d8f_bc36f48f2a2f.slice/crio-c5ea3c2fbdfea8a022d8becd6152dc48c8a73d7c332ae21cbd854671be2e2be2 WatchSource:0}: Error finding container c5ea3c2fbdfea8a022d8becd6152dc48c8a73d7c332ae21cbd854671be2e2be2: Status 404 returned error can't find the container with id c5ea3c2fbdfea8a022d8becd6152dc48c8a73d7c332ae21cbd854671be2e2be2
Oct 30 22:16:50.940258 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:50.940227    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-2b1e88-6b57666d9d-7lhjc" event={"ID":"c67bfb3e-cc4c-40a2-9d8f-bc36f48f2a2f","Type":"ContainerStarted","Data":"c5ea3c2fbdfea8a022d8becd6152dc48c8a73d7c332ae21cbd854671be2e2be2"}
Oct 30 22:16:53.219584 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:53.219542    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-d52c59-bbfdfb8dd-d7cx7"]
Oct 30 22:16:53.223422 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:53.223400    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d52c59-bbfdfb8dd-d7cx7"
Oct 30 22:16:53.237955 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:53.237935    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-d52c59-serving-cert\""
Oct 30 22:16:53.250026 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:53.249958    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-d52c59-bbfdfb8dd-d7cx7"]
Oct 30 22:16:53.300177 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:53.300151    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd421f77-d9c7-4bb8-833a-e9a861fa468f-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-d52c59-bbfdfb8dd-d7cx7\" (UID: \"dd421f77-d9c7-4bb8-833a-e9a861fa468f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d52c59-bbfdfb8dd-d7cx7"
Oct 30 22:16:53.300318 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:53.300274    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd421f77-d9c7-4bb8-833a-e9a861fa468f-metrics-certs\") pod \"maas-api-e2e-worker-w3-d52c59-bbfdfb8dd-d7cx7\" (UID: \"dd421f77-d9c7-4bb8-833a-e9a861fa468f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d52c59-bbfdfb8dd-d7cx7"
Oct 30 22:16:53.300385 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:53.300327    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxxkf\" (UniqueName: \"kubernetes.io/projected/dd421f77-d9c7-4bb8-833a-e9a861fa468f-kube-api-access-dxxkf\") pod \"maas-api-e2e-worker-w3-d52c59-bbfdfb8dd-d7cx7\" (UID: \"dd421f77-d9c7-4bb8-833a-e9a861fa468f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d52c59-bbfdfb8dd-d7cx7"
Oct 30 22:16:53.300449 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:53.300416    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd421f77-d9c7-4bb8-833a-e9a861fa468f-maas-api-tls\") pod \"maas-api-e2e-worker-w3-d52c59-bbfdfb8dd-d7cx7\" (UID: \"dd421f77-d9c7-4bb8-833a-e9a861fa468f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d52c59-bbfdfb8dd-d7cx7"
Oct 30 22:16:53.401923 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:53.401899    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd421f77-d9c7-4bb8-833a-e9a861fa468f-metrics-certs\") pod \"maas-api-e2e-worker-w3-d52c59-bbfdfb8dd-d7cx7\" (UID: \"dd421f77-d9c7-4bb8-833a-e9a861fa468f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d52c59-bbfdfb8dd-d7cx7"
Oct 30 22:16:53.402081 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:53.401946    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxxkf\" (UniqueName: \"kubernetes.io/projected/dd421f77-d9c7-4bb8-833a-e9a861fa468f-kube-api-access-dxxkf\") pod \"maas-api-e2e-worker-w3-d52c59-bbfdfb8dd-d7cx7\" (UID: \"dd421f77-d9c7-4bb8-833a-e9a861fa468f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d52c59-bbfdfb8dd-d7cx7"
Oct 30 22:16:53.402081 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:53.401985    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd421f77-d9c7-4bb8-833a-e9a861fa468f-maas-api-tls\") pod \"maas-api-e2e-worker-w3-d52c59-bbfdfb8dd-d7cx7\" (UID: \"dd421f77-d9c7-4bb8-833a-e9a861fa468f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d52c59-bbfdfb8dd-d7cx7"
Oct 30 22:16:53.402081 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:53.402034    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd421f77-d9c7-4bb8-833a-e9a861fa468f-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-d52c59-bbfdfb8dd-d7cx7\" (UID: \"dd421f77-d9c7-4bb8-833a-e9a861fa468f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d52c59-bbfdfb8dd-d7cx7"
Oct 30 22:16:53.402260 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:16:53.402117    4477 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w3-d52c59-serving-cert: secret "maas-api-e2e-worker-w3-d52c59-serving-cert" not found
Oct 30 22:16:53.402260 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:16:53.402175    4477 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd421f77-d9c7-4bb8-833a-e9a861fa468f-maas-api-tls podName:dd421f77-d9c7-4bb8-833a-e9a861fa468f nodeName:}" failed. No retries permitted until 2026-10-30 22:16:53.902159272 +0000 UTC m=+597.828085475 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/dd421f77-d9c7-4bb8-833a-e9a861fa468f-maas-api-tls") pod "maas-api-e2e-worker-w3-d52c59-bbfdfb8dd-d7cx7" (UID: "dd421f77-d9c7-4bb8-833a-e9a861fa468f") : secret "maas-api-e2e-worker-w3-d52c59-serving-cert" not found
Oct 30 22:16:53.402856 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:53.402824    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd421f77-d9c7-4bb8-833a-e9a861fa468f-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-d52c59-bbfdfb8dd-d7cx7\" (UID: \"dd421f77-d9c7-4bb8-833a-e9a861fa468f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d52c59-bbfdfb8dd-d7cx7"
Oct 30 22:16:53.404578 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:53.404558    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd421f77-d9c7-4bb8-833a-e9a861fa468f-metrics-certs\") pod \"maas-api-e2e-worker-w3-d52c59-bbfdfb8dd-d7cx7\" (UID: \"dd421f77-d9c7-4bb8-833a-e9a861fa468f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d52c59-bbfdfb8dd-d7cx7"
Oct 30 22:16:53.422341 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:53.422316    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxxkf\" (UniqueName: \"kubernetes.io/projected/dd421f77-d9c7-4bb8-833a-e9a861fa468f-kube-api-access-dxxkf\") pod \"maas-api-e2e-worker-w3-d52c59-bbfdfb8dd-d7cx7\" (UID: \"dd421f77-d9c7-4bb8-833a-e9a861fa468f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d52c59-bbfdfb8dd-d7cx7"
Oct 30 22:16:53.906267 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:53.906237    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd421f77-d9c7-4bb8-833a-e9a861fa468f-maas-api-tls\") pod \"maas-api-e2e-worker-w3-d52c59-bbfdfb8dd-d7cx7\" (UID: \"dd421f77-d9c7-4bb8-833a-e9a861fa468f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d52c59-bbfdfb8dd-d7cx7"
Oct 30 22:16:53.908635 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:53.908607    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd421f77-d9c7-4bb8-833a-e9a861fa468f-maas-api-tls\") pod \"maas-api-e2e-worker-w3-d52c59-bbfdfb8dd-d7cx7\" (UID: \"dd421f77-d9c7-4bb8-833a-e9a861fa468f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d52c59-bbfdfb8dd-d7cx7"
Oct 30 22:16:53.958259 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:53.958230    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-2b1e88-6b57666d9d-7lhjc" event={"ID":"c67bfb3e-cc4c-40a2-9d8f-bc36f48f2a2f","Type":"ContainerStarted","Data":"da2e50f1e9a9c0c4566dcdacf20718471f0e990d6d6c8177a08d176c3f42338e"}
Oct 30 22:16:53.958420 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:53.958403    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-2b1e88-6b57666d9d-7lhjc"
Oct 30 22:16:53.996836 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:53.996782    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-2b1e88-6b57666d9d-7lhjc" podStartSLOduration=2.775070362 podStartE2EDuration="4.99676565s" totalImagesPullingTime="2.221695288s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:16:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:16:53.993681569 +0000 UTC m=+597.919607776" watchObservedRunningTime="2026-10-30 22:16:53.99676565 +0000 UTC m=+597.922691857"
Oct 30 22:16:54.136015 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:54.135983    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d52c59-bbfdfb8dd-d7cx7"
Oct 30 22:16:54.431586 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:54.431518    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-d52c59-bbfdfb8dd-d7cx7"]
Oct 30 22:16:54.436615 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:16:54.436587    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd421f77_d9c7_4bb8_833a_e9a861fa468f.slice/crio-682490b0adfa5bc278e84fe5be9b87d6e17ca83d362adca264d0506f210af21d WatchSource:0}: Error finding container 682490b0adfa5bc278e84fe5be9b87d6e17ca83d362adca264d0506f210af21d: Status 404 returned error can't find the container with id 682490b0adfa5bc278e84fe5be9b87d6e17ca83d362adca264d0506f210af21d
Oct 30 22:16:54.748914 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:54.748842    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6bz92"]
Oct 30 22:16:54.965266 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:54.965230    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d52c59-bbfdfb8dd-d7cx7" event={"ID":"dd421f77-d9c7-4bb8-833a-e9a861fa468f","Type":"ContainerStarted","Data":"22ea87c9a8551f2c9dadaadb980faf2ef7f458fd368e7ef903cbdbfc37fb6b0f"}
Oct 30 22:16:54.965459 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:54.965284    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d52c59-bbfdfb8dd-d7cx7" event={"ID":"dd421f77-d9c7-4bb8-833a-e9a861fa468f","Type":"ContainerStarted","Data":"682490b0adfa5bc278e84fe5be9b87d6e17ca83d362adca264d0506f210af21d"}
Oct 30 22:16:55.007617 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:55.007526    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d52c59-bbfdfb8dd-d7cx7" podStartSLOduration=1.689564291 podStartE2EDuration="2.007511248s" totalImagesPullingTime="317.946957ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:16:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:16:55.006985011 +0000 UTC m=+598.932911226" watchObservedRunningTime="2026-10-30 22:16:55.007511248 +0000 UTC m=+598.933437453"
Oct 30 22:16:55.970622 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:55.970591    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d52c59-bbfdfb8dd-d7cx7"
Oct 30 22:16:56.245450 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:56.245386    4477 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jcf8l_015f218d-bf1f-4e7e-bb5d-b805af003246/ovn-acl-logging/0.log"
Oct 30 22:16:56.247446 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:56.247425    4477 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jcf8l_015f218d-bf1f-4e7e-bb5d-b805af003246/ovn-acl-logging/0.log"
Oct 30 22:16:56.266811 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:16:56.266778    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 22:16:58.865583 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:58.865549    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-012ba0-798cb567cc-k8959"]
Oct 30 22:16:58.869911 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:58.869891    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-012ba0-798cb567cc-k8959"
Oct 30 22:16:58.883826 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:58.883807    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w6-012ba0-serving-cert\""
Oct 30 22:16:58.895765 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:58.895742    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-012ba0-798cb567cc-k8959"]
Oct 30 22:16:58.949237 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:58.949208    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/130008a5-8f15-4ded-b8bd-d1903cb436b2-metrics-certs\") pod \"maas-api-e2e-worker-w6-012ba0-798cb567cc-k8959\" (UID: \"130008a5-8f15-4ded-b8bd-d1903cb436b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-012ba0-798cb567cc-k8959"
Oct 30 22:16:58.949375 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:58.949252    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jn86p\" (UniqueName: \"kubernetes.io/projected/130008a5-8f15-4ded-b8bd-d1903cb436b2-kube-api-access-jn86p\") pod \"maas-api-e2e-worker-w6-012ba0-798cb567cc-k8959\" (UID: \"130008a5-8f15-4ded-b8bd-d1903cb436b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-012ba0-798cb567cc-k8959"
Oct 30 22:16:58.949375 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:58.949324    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/130008a5-8f15-4ded-b8bd-d1903cb436b2-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-012ba0-798cb567cc-k8959\" (UID: \"130008a5-8f15-4ded-b8bd-d1903cb436b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-012ba0-798cb567cc-k8959"
Oct 30 22:16:58.949375 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:58.949345    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/130008a5-8f15-4ded-b8bd-d1903cb436b2-maas-api-tls\") pod \"maas-api-e2e-worker-w6-012ba0-798cb567cc-k8959\" (UID: \"130008a5-8f15-4ded-b8bd-d1903cb436b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-012ba0-798cb567cc-k8959"
Oct 30 22:16:59.050864 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:59.050835    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/130008a5-8f15-4ded-b8bd-d1903cb436b2-metrics-certs\") pod \"maas-api-e2e-worker-w6-012ba0-798cb567cc-k8959\" (UID: \"130008a5-8f15-4ded-b8bd-d1903cb436b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-012ba0-798cb567cc-k8959"
Oct 30 22:16:59.051025 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:59.050903    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jn86p\" (UniqueName: \"kubernetes.io/projected/130008a5-8f15-4ded-b8bd-d1903cb436b2-kube-api-access-jn86p\") pod \"maas-api-e2e-worker-w6-012ba0-798cb567cc-k8959\" (UID: \"130008a5-8f15-4ded-b8bd-d1903cb436b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-012ba0-798cb567cc-k8959"
Oct 30 22:16:59.051025 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:59.050939    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/130008a5-8f15-4ded-b8bd-d1903cb436b2-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-012ba0-798cb567cc-k8959\" (UID: \"130008a5-8f15-4ded-b8bd-d1903cb436b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-012ba0-798cb567cc-k8959"
Oct 30 22:16:59.051025 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:59.050965    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/130008a5-8f15-4ded-b8bd-d1903cb436b2-maas-api-tls\") pod \"maas-api-e2e-worker-w6-012ba0-798cb567cc-k8959\" (UID: \"130008a5-8f15-4ded-b8bd-d1903cb436b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-012ba0-798cb567cc-k8959"
Oct 30 22:16:59.051703 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:59.051671    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/130008a5-8f15-4ded-b8bd-d1903cb436b2-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-012ba0-798cb567cc-k8959\" (UID: \"130008a5-8f15-4ded-b8bd-d1903cb436b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-012ba0-798cb567cc-k8959"
Oct 30 22:16:59.053618 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:59.053595    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/130008a5-8f15-4ded-b8bd-d1903cb436b2-maas-api-tls\") pod \"maas-api-e2e-worker-w6-012ba0-798cb567cc-k8959\" (UID: \"130008a5-8f15-4ded-b8bd-d1903cb436b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-012ba0-798cb567cc-k8959"
Oct 30 22:16:59.053618 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:59.053613    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/130008a5-8f15-4ded-b8bd-d1903cb436b2-metrics-certs\") pod \"maas-api-e2e-worker-w6-012ba0-798cb567cc-k8959\" (UID: \"130008a5-8f15-4ded-b8bd-d1903cb436b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-012ba0-798cb567cc-k8959"
Oct 30 22:16:59.073023 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:59.072998    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jn86p\" (UniqueName: \"kubernetes.io/projected/130008a5-8f15-4ded-b8bd-d1903cb436b2-kube-api-access-jn86p\") pod \"maas-api-e2e-worker-w6-012ba0-798cb567cc-k8959\" (UID: \"130008a5-8f15-4ded-b8bd-d1903cb436b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-012ba0-798cb567cc-k8959"
Oct 30 22:16:59.182215 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:59.182180    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-012ba0-798cb567cc-k8959"
Oct 30 22:16:59.474827 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:59.474765    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-012ba0-798cb567cc-k8959"]
Oct 30 22:16:59.479938 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:16:59.479913    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod130008a5_8f15_4ded_b8bd_d1903cb436b2.slice/crio-e9f8871af29e9291c6f000a10ce55ee482bc42a79a5fee6d781bc12b49bae425 WatchSource:0}: Error finding container e9f8871af29e9291c6f000a10ce55ee482bc42a79a5fee6d781bc12b49bae425: Status 404 returned error can't find the container with id e9f8871af29e9291c6f000a10ce55ee482bc42a79a5fee6d781bc12b49bae425
Oct 30 22:16:59.481363 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:59.481341    4477 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 22:16:59.970525 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:59.970503    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-2b1e88-6b57666d9d-7lhjc"
Oct 30 22:16:59.994334 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:59.994304    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-012ba0-798cb567cc-k8959" event={"ID":"130008a5-8f15-4ded-b8bd-d1903cb436b2","Type":"ContainerStarted","Data":"7c3e38a794382bbf21dc51ce6eb57f9c087a5baebdf5300880aed557eb5ca4ec"}
Oct 30 22:16:59.994452 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:59.994353    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-012ba0-798cb567cc-k8959" event={"ID":"130008a5-8f15-4ded-b8bd-d1903cb436b2","Type":"ContainerStarted","Data":"e9f8871af29e9291c6f000a10ce55ee482bc42a79a5fee6d781bc12b49bae425"}
Oct 30 22:16:59.994452 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:16:59.994442    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-012ba0-798cb567cc-k8959"
Oct 30 22:17:00.044357 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:17:00.044306    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-012ba0-798cb567cc-k8959" podStartSLOduration=1.715959875 podStartE2EDuration="2.044292003s" totalImagesPullingTime="328.332128ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:16:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:17:00.043035577 +0000 UTC m=+603.968961783" watchObservedRunningTime="2026-10-30 22:17:00.044292003 +0000 UTC m=+603.970218231"
Oct 30 22:17:01.936525 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:17:01.936489    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6bz92"]
Oct 30 22:17:01.985161 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:17:01.985139    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d52c59-bbfdfb8dd-d7cx7"
Oct 30 22:17:06.005076 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:17:06.005032    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-012ba0-798cb567cc-k8959"
Oct 30 22:17:11.335132 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:17:11.335089    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6bz92"]
Oct 30 22:17:23.671391 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:17:23.671346    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-2b1e88/simulator-w1-2b1e88-kserve-6c579b99f8-q5wpq"]
Oct 30 22:17:23.685568 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:17:23.685526    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-2b1e88/simulator-w1-2b1e88-kserve-6c579b99f8-q5wpq"
Oct 30 22:17:23.695285 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:17:23.695258    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-2b1e88/simulator-w1-2b1e88-kserve-6c579b99f8-q5wpq"]
Oct 30 22:17:23.703132 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:17:23.703108    4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-2b1e88\"/\"kube-root-ca.crt\""
Oct 30 22:17:23.703247 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:17:23.703168    4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-2b1e88\"/\"openshift-service-ca.crt\""
Oct 30 22:17:23.708289 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:17:23.708274    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-2b1e88\"/\"default-dockercfg-pm7m5\""
Oct 30 22:17:23.708451 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:17:23.708429    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-2b1e88\"/\"simulator-w1-2b1e88-kserve-self-signed-certs\""
Oct 30 22:17:23.786752 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:17:23.786719    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/722f10d8-b7a2-4bf7-a274-9669529ef427-tmp-dir\") pod \"simulator-w1-2b1e88-kserve-6c579b99f8-q5wpq\" (UID: \"722f10d8-b7a2-4bf7-a274-9669529ef427\") " pod="e2e-models-e2e-worker-w1-2b1e88/simulator-w1-2b1e88-kserve-6c579b99f8-q5wpq"
Oct 30 22:17:23.786890 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:17:23.786820    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/722f10d8-b7a2-4bf7-a274-9669529ef427-model-cache\") pod \"simulator-w1-2b1e88-kserve-6c579b99f8-q5wpq\" (UID: \"722f10d8-b7a2-4bf7-a274-9669529ef427\") " pod="e2e-models-e2e-worker-w1-2b1e88/simulator-w1-2b1e88-kserve-6c579b99f8-q5wpq"
Oct 30 22:17:23.786936 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:17:23.786898    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/722f10d8-b7a2-4bf7-a274-9669529ef427-tls-certs\") pod \"simulator-w1-2b1e88-kserve-6c579b99f8-q5wpq\" (UID: \"722f10d8-b7a2-4bf7-a274-9669529ef427\") " pod="e2e-models-e2e-worker-w1-2b1e88/simulator-w1-2b1e88-kserve-6c579b99f8-q5wpq"
Oct 30 22:17:23.786996 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:17:23.786976    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/722f10d8-b7a2-4bf7-a274-9669529ef427-home\") pod \"simulator-w1-2b1e88-kserve-6c579b99f8-q5wpq\" (UID: \"722f10d8-b7a2-4bf7-a274-9669529ef427\") " pod="e2e-models-e2e-worker-w1-2b1e88/simulator-w1-2b1e88-kserve-6c579b99f8-q5wpq"
Oct 30 22:17:23.787030 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:17:23.787014    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6qdg\" (UniqueName: \"kubernetes.io/projected/722f10d8-b7a2-4bf7-a274-9669529ef427-kube-api-access-p6qdg\") pod \"simulator-w1-2b1e88-kserve-6c579b99f8-q5wpq\" (UID: \"722f10d8-b7a2-4bf7-a274-9669529ef427\") " pod="e2e-models-e2e-worker-w1-2b1e88/simulator-w1-2b1e88-kserve-6c579b99f8-q5wpq"
Oct 30 22:17:23.787083 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:17:23.787059    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/722f10d8-b7a2-4bf7-a274-9669529ef427-dshm\") pod \"simulator-w1-2b1e88-kserve-6c579b99f8-q5wpq\" (UID: \"722f10d8-b7a2-4bf7-a274-9669529ef427\") " pod="e2e-models-e2e-worker-w1-2b1e88/simulator-w1-2b1e88-kserve-6c579b99f8-q5wpq"
Oct 30 22:17:23.887688 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:17:23.887659    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6qdg\" (UniqueName: \"kubernetes.io/projected/722f10d8-b7a2-4bf7-a274-9669529ef427-kube-api-access-p6qdg\") pod \"simulator-w1-2b1e88-kserve-6c579b99f8-q5wpq\" (UID: \"722f10d8-b7a2-4bf7-a274-9669529ef427\") " pod="e2e-models-e2e-worker-w1-2b1e88/simulator-w1-2b1e88-kserve-6c579b99f8-q5wpq"
Oct 30 22:17:23.887862 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:17:23.887693    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/722f10d8-b7a2-4bf7-a274-9669529ef427-dshm\") pod \"simulator-w1-2b1e88-kserve-6c579b99f8-q5wpq\" (UID: \"722f10d8-b7a2-4bf7-a274-9669529ef427\") " pod="e2e-models-e2e-worker-w1-2b1e88/simulator-w1-2b1e88-kserve-6c579b99f8-q5wpq"
Oct 30 22:17:23.887935 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:17:23.887896    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/722f10d8-b7a2-4bf7-a274-9669529ef427-tmp-dir\") pod \"simulator-w1-2b1e88-kserve-6c579b99f8-q5wpq\" (UID: \"722f10d8-b7a2-4bf7-a274-9669529ef427\") " pod="e2e-models-e2e-worker-w1-2b1e88/simulator-w1-2b1e88-kserve-6c579b99f8-q5wpq"
Oct 30 22:17:23.887990 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:17:23.887951    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/722f10d8-b7a2-4bf7-a274-9669529ef427-model-cache\") pod \"simulator-w1-2b1e88-kserve-6c579b99f8-q5wpq\" (UID: \"722f10d8-b7a2-4bf7-a274-9669529ef427\") " pod="e2e-models-e2e-worker-w1-2b1e88/simulator-w1-2b1e88-kserve-6c579b99f8-q5wpq"
Oct 30 22:17:23.888033 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:17:23.888015    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/722f10d8-b7a2-4bf7-a274-9669529ef427-tls-certs\") pod \"simulator-w1-2b1e88-kserve-6c579b99f8-q5wpq\" (UID: \"722f10d8-b7a2-4bf7-a274-9669529ef427\") " pod="e2e-models-e2e-worker-w1-2b1e88/simulator-w1-2b1e88-kserve-6c579b99f8-q5wpq"
Oct 30 22:17:23.888098 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:17:23.888076    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/722f10d8-b7a2-4bf7-a274-9669529ef427-home\") pod \"simulator-w1-2b1e88-kserve-6c579b99f8-q5wpq\" (UID: \"722f10d8-b7a2-4bf7-a274-9669529ef427\") " pod="e2e-models-e2e-worker-w1-2b1e88/simulator-w1-2b1e88-kserve-6c579b99f8-q5wpq"
Oct 30 22:17:23.888366 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:17:23.888338    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/722f10d8-b7a2-4bf7-a274-9669529ef427-tmp-dir\") pod \"simulator-w1-2b1e88-kserve-6c579b99f8-q5wpq\" (UID: \"722f10d8-b7a2-4bf7-a274-9669529ef427\") " pod="e2e-models-e2e-worker-w1-2b1e88/simulator-w1-2b1e88-kserve-6c579b99f8-q5wpq"
Oct 30 22:17:23.888454 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:17:23.888345    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/722f10d8-b7a2-4bf7-a274-9669529ef427-model-cache\") pod \"simulator-w1-2b1e88-kserve-6c579b99f8-q5wpq\" (UID: \"722f10d8-b7a2-4bf7-a274-9669529ef427\") " pod="e2e-models-e2e-worker-w1-2b1e88/simulator-w1-2b1e88-kserve-6c579b99f8-q5wpq"
Oct 30 22:17:23.888454 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:17:23.888392    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/722f10d8-b7a2-4bf7-a274-9669529ef427-home\") pod \"simulator-w1-2b1e88-kserve-6c579b99f8-q5wpq\" (UID: \"722f10d8-b7a2-4bf7-a274-9669529ef427\") " pod="e2e-models-e2e-worker-w1-2b1e88/simulator-w1-2b1e88-kserve-6c579b99f8-q5wpq"
Oct 30 22:17:23.890007 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:17:23.889981    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/722f10d8-b7a2-4bf7-a274-9669529ef427-dshm\") pod \"simulator-w1-2b1e88-kserve-6c579b99f8-q5wpq\" (UID: \"722f10d8-b7a2-4bf7-a274-9669529ef427\") " pod="e2e-models-e2e-worker-w1-2b1e88/simulator-w1-2b1e88-kserve-6c579b99f8-q5wpq"
Oct 30 22:17:23.890336 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:17:23.890320    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/722f10d8-b7a2-4bf7-a274-9669529ef427-tls-certs\") pod \"simulator-w1-2b1e88-kserve-6c579b99f8-q5wpq\" (UID: \"722f10d8-b7a2-4bf7-a274-9669529ef427\") " pod="e2e-models-e2e-worker-w1-2b1e88/simulator-w1-2b1e88-kserve-6c579b99f8-q5wpq"
Oct 30 22:17:23.904973 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:17:23.904954    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6qdg\" (UniqueName: \"kubernetes.io/projected/722f10d8-b7a2-4bf7-a274-9669529ef427-kube-api-access-p6qdg\") pod \"simulator-w1-2b1e88-kserve-6c579b99f8-q5wpq\" (UID: \"722f10d8-b7a2-4bf7-a274-9669529ef427\") " pod="e2e-models-e2e-worker-w1-2b1e88/simulator-w1-2b1e88-kserve-6c579b99f8-q5wpq"
Oct 30 22:17:24.000923 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:17:24.000854    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-2b1e88/simulator-w1-2b1e88-kserve-6c579b99f8-q5wpq"
Oct 30 22:17:24.295206 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:17:24.295125    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-2b1e88/simulator-w1-2b1e88-kserve-6c579b99f8-q5wpq"]
Oct 30 22:17:24.300457 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:17:24.300432    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod722f10d8_b7a2_4bf7_a274_9669529ef427.slice/crio-946df0cdeb6afa4a96955a531dcb5a150317269b7cffff39f0b83e1849f759d1 WatchSource:0}: Error finding container 946df0cdeb6afa4a96955a531dcb5a150317269b7cffff39f0b83e1849f759d1: Status 404 returned error can't find the container with id 946df0cdeb6afa4a96955a531dcb5a150317269b7cffff39f0b83e1849f759d1
Oct 30 22:17:24.831752 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:17:24.831701    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6bz92"]
Oct 30 22:17:25.123835 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:17:25.123803    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-2b1e88/simulator-w1-2b1e88-kserve-6c579b99f8-q5wpq" event={"ID":"722f10d8-b7a2-4bf7-a274-9669529ef427","Type":"ContainerStarted","Data":"946df0cdeb6afa4a96955a531dcb5a150317269b7cffff39f0b83e1849f759d1"}
Oct 30 22:17:26.129871 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:17:26.129841    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-2b1e88/simulator-w1-2b1e88-kserve-6c579b99f8-q5wpq" event={"ID":"722f10d8-b7a2-4bf7-a274-9669529ef427","Type":"ContainerStarted","Data":"bafc7cd2e69bc49eba568e173fdf9c3499aefad1893df48c60d0ed73a7600ffb"}
Oct 30 22:17:26.166904 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:17:26.166860    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-2b1e88/simulator-w1-2b1e88-kserve-6c579b99f8-q5wpq" podStartSLOduration=2.142491569 podStartE2EDuration="3.16684716s" totalImagesPullingTime="1.024355591s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:17:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:17:26.166154473 +0000 UTC m=+630.092080678" watchObservedRunningTime="2026-10-30 22:17:26.16684716 +0000 UTC m=+630.092773366"
Oct 30 22:17:26.265454 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:17:26.265421    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 22:17:27.001977 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:17:27.001950    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-2b1e88/simulator-w1-2b1e88-kserve-6c579b99f8-q5wpq"
Oct 30 22:17:34.001233 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:17:34.001201    4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-2b1e88/simulator-w1-2b1e88-kserve-6c579b99f8-q5wpq"
Oct 30 22:17:34.014333 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:17:34.014305    4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-2b1e88/simulator-w1-2b1e88-kserve-6c579b99f8-q5wpq"
Oct 30 22:17:34.180329 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:17:34.180301    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-2b1e88/simulator-w1-2b1e88-kserve-6c579b99f8-q5wpq"
Oct 30 22:17:42.542733 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:17:42.542659    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6bz92"]
Oct 30 22:17:56.269262 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:17:56.269221    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 22:17:58.448142 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:17:58.448105    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6bz92"]
Oct 30 22:18:00.478314 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:18:00.478281    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-54114e/premium-w5-54114e-kserve-57f6747bcc-7wjrw"]
Oct 30 22:18:00.485423 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:18:00.485401    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-54114e/premium-w5-54114e-kserve-57f6747bcc-7wjrw"
Oct 30 22:18:00.503143 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:18:00.502989    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-54114e/premium-w5-54114e-kserve-57f6747bcc-7wjrw"]
Oct 30 22:18:00.510794 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:18:00.510771    4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-54114e\"/\"openshift-service-ca.crt\""
Oct 30 22:18:00.510909 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:18:00.510771    4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-54114e\"/\"kube-root-ca.crt\""
Oct 30 22:18:00.510909 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:18:00.510772    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-54114e\"/\"default-dockercfg-nnn5d\""
Oct 30 22:18:00.511208 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:18:00.511187    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-54114e\"/\"premium-w5-54114e-kserve-self-signed-certs\""
Oct 30 22:18:00.620192 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:18:00.620157    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8tk4\" (UniqueName: \"kubernetes.io/projected/61990d22-ab8f-45bc-a4d6-4c8d528742f1-kube-api-access-j8tk4\") pod \"premium-w5-54114e-kserve-57f6747bcc-7wjrw\" (UID: \"61990d22-ab8f-45bc-a4d6-4c8d528742f1\") " pod="e2e-models-e2e-worker-w5-54114e/premium-w5-54114e-kserve-57f6747bcc-7wjrw"
Oct 30 22:18:00.620386 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:18:00.620240    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61990d22-ab8f-45bc-a4d6-4c8d528742f1-dshm\") pod \"premium-w5-54114e-kserve-57f6747bcc-7wjrw\" (UID: \"61990d22-ab8f-45bc-a4d6-4c8d528742f1\") " pod="e2e-models-e2e-worker-w5-54114e/premium-w5-54114e-kserve-57f6747bcc-7wjrw"
Oct 30 22:18:00.620386 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:18:00.620334    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61990d22-ab8f-45bc-a4d6-4c8d528742f1-home\") pod \"premium-w5-54114e-kserve-57f6747bcc-7wjrw\" (UID: \"61990d22-ab8f-45bc-a4d6-4c8d528742f1\") " pod="e2e-models-e2e-worker-w5-54114e/premium-w5-54114e-kserve-57f6747bcc-7wjrw"
Oct 30 22:18:00.620508 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:18:00.620385    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61990d22-ab8f-45bc-a4d6-4c8d528742f1-tmp-dir\") pod \"premium-w5-54114e-kserve-57f6747bcc-7wjrw\" (UID: \"61990d22-ab8f-45bc-a4d6-4c8d528742f1\") " pod="e2e-models-e2e-worker-w5-54114e/premium-w5-54114e-kserve-57f6747bcc-7wjrw"
Oct 30 22:18:00.620508 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:18:00.620456    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61990d22-ab8f-45bc-a4d6-4c8d528742f1-tls-certs\") pod \"premium-w5-54114e-kserve-57f6747bcc-7wjrw\" (UID: \"61990d22-ab8f-45bc-a4d6-4c8d528742f1\") " pod="e2e-models-e2e-worker-w5-54114e/premium-w5-54114e-kserve-57f6747bcc-7wjrw"
Oct 30 22:18:00.620508 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:18:00.620504    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61990d22-ab8f-45bc-a4d6-4c8d528742f1-model-cache\") pod \"premium-w5-54114e-kserve-57f6747bcc-7wjrw\" (UID: \"61990d22-ab8f-45bc-a4d6-4c8d528742f1\") " pod="e2e-models-e2e-worker-w5-54114e/premium-w5-54114e-kserve-57f6747bcc-7wjrw"
Oct 30 22:18:00.721561 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:18:00.721529    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61990d22-ab8f-45bc-a4d6-4c8d528742f1-dshm\") pod \"premium-w5-54114e-kserve-57f6747bcc-7wjrw\" (UID: \"61990d22-ab8f-45bc-a4d6-4c8d528742f1\") " pod="e2e-models-e2e-worker-w5-54114e/premium-w5-54114e-kserve-57f6747bcc-7wjrw"
Oct 30 22:18:00.721747 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:18:00.721581    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61990d22-ab8f-45bc-a4d6-4c8d528742f1-home\") pod \"premium-w5-54114e-kserve-57f6747bcc-7wjrw\" (UID: \"61990d22-ab8f-45bc-a4d6-4c8d528742f1\") " pod="e2e-models-e2e-worker-w5-54114e/premium-w5-54114e-kserve-57f6747bcc-7wjrw"
Oct 30 22:18:00.721747 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:18:00.721618    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61990d22-ab8f-45bc-a4d6-4c8d528742f1-tmp-dir\") pod \"premium-w5-54114e-kserve-57f6747bcc-7wjrw\" (UID: \"61990d22-ab8f-45bc-a4d6-4c8d528742f1\") " pod="e2e-models-e2e-worker-w5-54114e/premium-w5-54114e-kserve-57f6747bcc-7wjrw"
Oct 30 22:18:00.721747 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:18:00.721646    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61990d22-ab8f-45bc-a4d6-4c8d528742f1-tls-certs\") pod \"premium-w5-54114e-kserve-57f6747bcc-7wjrw\" (UID: \"61990d22-ab8f-45bc-a4d6-4c8d528742f1\") " pod="e2e-models-e2e-worker-w5-54114e/premium-w5-54114e-kserve-57f6747bcc-7wjrw"
Oct 30 22:18:00.721917 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:18:00.721803    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61990d22-ab8f-45bc-a4d6-4c8d528742f1-model-cache\") pod \"premium-w5-54114e-kserve-57f6747bcc-7wjrw\" (UID: \"61990d22-ab8f-45bc-a4d6-4c8d528742f1\") " pod="e2e-models-e2e-worker-w5-54114e/premium-w5-54114e-kserve-57f6747bcc-7wjrw"
Oct 30 22:18:00.721917 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:18:00.721888    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8tk4\" (UniqueName: \"kubernetes.io/projected/61990d22-ab8f-45bc-a4d6-4c8d528742f1-kube-api-access-j8tk4\") pod \"premium-w5-54114e-kserve-57f6747bcc-7wjrw\" (UID: \"61990d22-ab8f-45bc-a4d6-4c8d528742f1\") " pod="e2e-models-e2e-worker-w5-54114e/premium-w5-54114e-kserve-57f6747bcc-7wjrw"
Oct 30 22:18:00.722181 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:18:00.722155    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61990d22-ab8f-45bc-a4d6-4c8d528742f1-tmp-dir\") pod \"premium-w5-54114e-kserve-57f6747bcc-7wjrw\" (UID: \"61990d22-ab8f-45bc-a4d6-4c8d528742f1\") " pod="e2e-models-e2e-worker-w5-54114e/premium-w5-54114e-kserve-57f6747bcc-7wjrw"
Oct 30 22:18:00.722288 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:18:00.722217    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61990d22-ab8f-45bc-a4d6-4c8d528742f1-model-cache\") pod \"premium-w5-54114e-kserve-57f6747bcc-7wjrw\" (UID: \"61990d22-ab8f-45bc-a4d6-4c8d528742f1\") " pod="e2e-models-e2e-worker-w5-54114e/premium-w5-54114e-kserve-57f6747bcc-7wjrw"
Oct 30 22:18:00.722350 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:18:00.722298    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61990d22-ab8f-45bc-a4d6-4c8d528742f1-home\") pod \"premium-w5-54114e-kserve-57f6747bcc-7wjrw\" (UID: \"61990d22-ab8f-45bc-a4d6-4c8d528742f1\") " pod="e2e-models-e2e-worker-w5-54114e/premium-w5-54114e-kserve-57f6747bcc-7wjrw"
Oct 30 22:18:00.724008 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:18:00.723975    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61990d22-ab8f-45bc-a4d6-4c8d528742f1-dshm\") pod \"premium-w5-54114e-kserve-57f6747bcc-7wjrw\" (UID: \"61990d22-ab8f-45bc-a4d6-4c8d528742f1\") " pod="e2e-models-e2e-worker-w5-54114e/premium-w5-54114e-kserve-57f6747bcc-7wjrw"
Oct 30 22:18:00.724423 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:18:00.724405    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61990d22-ab8f-45bc-a4d6-4c8d528742f1-tls-certs\") pod \"premium-w5-54114e-kserve-57f6747bcc-7wjrw\" (UID: \"61990d22-ab8f-45bc-a4d6-4c8d528742f1\") " pod="e2e-models-e2e-worker-w5-54114e/premium-w5-54114e-kserve-57f6747bcc-7wjrw"
Oct 30 22:18:00.739016 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:18:00.738965    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8tk4\" (UniqueName: \"kubernetes.io/projected/61990d22-ab8f-45bc-a4d6-4c8d528742f1-kube-api-access-j8tk4\") pod \"premium-w5-54114e-kserve-57f6747bcc-7wjrw\" (UID: \"61990d22-ab8f-45bc-a4d6-4c8d528742f1\") " pod="e2e-models-e2e-worker-w5-54114e/premium-w5-54114e-kserve-57f6747bcc-7wjrw"
Oct 30 22:18:00.801015 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:18:00.800987    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-54114e/premium-w5-54114e-kserve-57f6747bcc-7wjrw"
Oct 30 22:18:01.095671 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:18:01.095586    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-54114e/premium-w5-54114e-kserve-57f6747bcc-7wjrw"]
Oct 30 22:18:01.100162 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:18:01.100129    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod61990d22_ab8f_45bc_a4d6_4c8d528742f1.slice/crio-4dc4c217a29ce4f46a614ba265f7244b75cb30cff11eadbd39b1944a6075f67a WatchSource:0}: Error finding container 4dc4c217a29ce4f46a614ba265f7244b75cb30cff11eadbd39b1944a6075f67a: Status 404 returned error can't find the container with id 4dc4c217a29ce4f46a614ba265f7244b75cb30cff11eadbd39b1944a6075f67a
Oct 30 22:18:01.308399 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:18:01.308365    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-54114e/premium-w5-54114e-kserve-57f6747bcc-7wjrw" event={"ID":"61990d22-ab8f-45bc-a4d6-4c8d528742f1","Type":"ContainerStarted","Data":"823ec907da0c45ca9ffd6f1b97c7e7422a4e68fb5c62fdb6892da92cc7454e65"}
Oct 30 22:18:01.308399 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:18:01.308403    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-54114e/premium-w5-54114e-kserve-57f6747bcc-7wjrw" event={"ID":"61990d22-ab8f-45bc-a4d6-4c8d528742f1","Type":"ContainerStarted","Data":"4dc4c217a29ce4f46a614ba265f7244b75cb30cff11eadbd39b1944a6075f67a"}
Oct 30 22:18:01.344281 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:18:01.344239    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-54114e/premium-w5-54114e-kserve-57f6747bcc-7wjrw" podStartSLOduration=1.344226699 podStartE2EDuration="1.344226699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:18:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:18:01.342112627 +0000 UTC m=+665.268038857" watchObservedRunningTime="2026-10-30 22:18:01.344226699 +0000 UTC m=+665.270152905"
Oct 30 22:18:01.801584 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:18:01.801559    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-54114e/premium-w5-54114e-kserve-57f6747bcc-7wjrw"
Oct 30 22:18:09.995167 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:18:09.994947    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-0ad7c5/premium-w4-0ad7c5-kserve-7bbf78cd4-pm6wc"]
Oct 30 22:18:09.999941 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:18:09.999920    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0ad7c5/premium-w4-0ad7c5-kserve-7bbf78cd4-pm6wc"
Oct 30 22:18:10.025259 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:18:10.025223    4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-0ad7c5\"/\"openshift-service-ca.crt\""
Oct 30 22:18:10.025259 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:18:10.025246    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-0ad7c5\"/\"default-dockercfg-gn2fb\""
Oct 30 22:18:10.025437 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:18:10.025223    4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-0ad7c5\"/\"kube-root-ca.crt\""
Oct 30 22:18:10.025437 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:18:10.025223    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-0ad7c5\"/\"premium-w4-0ad7c5-kserve-self-signed-certs\""
Oct 30 22:18:10.027486 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:18:10.027466    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-0ad7c5/premium-w4-0ad7c5-kserve-7bbf78cd4-pm6wc"]
Oct 30 22:18:10.112432 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:18:10.112407    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c10371e-56e7-42ec-9ef8-0802e15cb2ff-model-cache\") pod \"premium-w4-0ad7c5-kserve-7bbf78cd4-pm6wc\" (UID: \"5c10371e-56e7-42ec-9ef8-0802e15cb2ff\") " pod="e2e-models-e2e-worker-w4-0ad7c5/premium-w4-0ad7c5-kserve-7bbf78cd4-pm6wc"
Oct 30 22:18:10.112606 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:18:10.112450    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c10371e-56e7-42ec-9ef8-0802e15cb2ff-tmp-dir\") pod \"premium-w4-0ad7c5-kserve-7bbf78cd4-pm6wc\" (UID: \"5c10371e-56e7-42ec-9ef8-0802e15cb2ff\") " pod="e2e-models-e2e-worker-w4-0ad7c5/premium-w4-0ad7c5-kserve-7bbf78cd4-pm6wc"
Oct 30 22:18:10.112606 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:18:10.112472    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c10371e-56e7-42ec-9ef8-0802e15cb2ff-home\") pod \"premium-w4-0ad7c5-kserve-7bbf78cd4-pm6wc\" (UID: \"5c10371e-56e7-42ec-9ef8-0802e15cb2ff\") " pod="e2e-models-e2e-worker-w4-0ad7c5/premium-w4-0ad7c5-kserve-7bbf78cd4-pm6wc"
Oct 30 22:18:10.112729 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:18:10.112599    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9l6s\" (UniqueName: \"kubernetes.io/projected/5c10371e-56e7-42ec-9ef8-0802e15cb2ff-kube-api-access-m9l6s\") pod \"premium-w4-0ad7c5-kserve-7bbf78cd4-pm6wc\" (UID: \"5c10371e-56e7-42ec-9ef8-0802e15cb2ff\") " pod="e2e-models-e2e-worker-w4-0ad7c5/premium-w4-0ad7c5-kserve-7bbf78cd4-pm6wc"
Oct 30 22:18:10.112729 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:18:10.112666    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c10371e-56e7-42ec-9ef8-0802e15cb2ff-tls-certs\") pod \"premium-w4-0ad7c5-kserve-7bbf78cd4-pm6wc\" (UID: \"5c10371e-56e7-42ec-9ef8-0802e15cb2ff\") " pod="e2e-models-e2e-worker-w4-0ad7c5/premium-w4-0ad7c5-kserve-7bbf78cd4-pm6wc"
Oct 30 22:18:10.112729 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:18:10.112695    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c10371e-56e7-42ec-9ef8-0802e15cb2ff-dshm\") pod \"premium-w4-0ad7c5-kserve-7bbf78cd4-pm6wc\" (UID: \"5c10371e-56e7-42ec-9ef8-0802e15cb2ff\") " pod="e2e-models-e2e-worker-w4-0ad7c5/premium-w4-0ad7c5-kserve-7bbf78cd4-pm6wc"
Oct 30 22:18:10.213809 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:18:10.213776    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c10371e-56e7-42ec-9ef8-0802e15cb2ff-model-cache\") pod \"premium-w4-0ad7c5-kserve-7bbf78cd4-pm6wc\" (UID: \"5c10371e-56e7-42ec-9ef8-0802e15cb2ff\") " pod="e2e-models-e2e-worker-w4-0ad7c5/premium-w4-0ad7c5-kserve-7bbf78cd4-pm6wc"
Oct 30 22:18:10.213809 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:18:10.213816    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c10371e-56e7-42ec-9ef8-0802e15cb2ff-tmp-dir\") pod \"premium-w4-0ad7c5-kserve-7bbf78cd4-pm6wc\" (UID: \"5c10371e-56e7-42ec-9ef8-0802e15cb2ff\") " pod="e2e-models-e2e-worker-w4-0ad7c5/premium-w4-0ad7c5-kserve-7bbf78cd4-pm6wc"
Oct 30 22:18:10.214036 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:18:10.213843    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c10371e-56e7-42ec-9ef8-0802e15cb2ff-home\") pod \"premium-w4-0ad7c5-kserve-7bbf78cd4-pm6wc\" (UID: \"5c10371e-56e7-42ec-9ef8-0802e15cb2ff\") " pod="e2e-models-e2e-worker-w4-0ad7c5/premium-w4-0ad7c5-kserve-7bbf78cd4-pm6wc"
Oct 30 22:18:10.214036 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:18:10.213870    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9l6s\" (UniqueName: \"kubernetes.io/projected/5c10371e-56e7-42ec-9ef8-0802e15cb2ff-kube-api-access-m9l6s\") pod \"premium-w4-0ad7c5-kserve-7bbf78cd4-pm6wc\" (UID: \"5c10371e-56e7-42ec-9ef8-0802e15cb2ff\") " pod="e2e-models-e2e-worker-w4-0ad7c5/premium-w4-0ad7c5-kserve-7bbf78cd4-pm6wc"
Oct 30 22:18:10.214036 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:18:10.213898    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c10371e-56e7-42ec-9ef8-0802e15cb2ff-tls-certs\") pod \"premium-w4-0ad7c5-kserve-7bbf78cd4-pm6wc\" (UID: \"5c10371e-56e7-42ec-9ef8-0802e15cb2ff\") " pod="e2e-models-e2e-worker-w4-0ad7c5/premium-w4-0ad7c5-kserve-7bbf78cd4-pm6wc"
Oct 30 22:18:10.214036 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:18:10.213932    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c10371e-56e7-42ec-9ef8-0802e15cb2ff-dshm\") pod \"premium-w4-0ad7c5-kserve-7bbf78cd4-pm6wc\" (UID: \"5c10371e-56e7-42ec-9ef8-0802e15cb2ff\") " pod="e2e-models-e2e-worker-w4-0ad7c5/premium-w4-0ad7c5-kserve-7bbf78cd4-pm6wc"
Oct 30 22:18:10.214383 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:18:10.214357    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c10371e-56e7-42ec-9ef8-0802e15cb2ff-model-cache\") pod \"premium-w4-0ad7c5-kserve-7bbf78cd4-pm6wc\" (UID: \"5c10371e-56e7-42ec-9ef8-0802e15cb2ff\") " pod="e2e-models-e2e-worker-w4-0ad7c5/premium-w4-0ad7c5-kserve-7bbf78cd4-pm6wc"
Oct 30 22:18:10.214455 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:18:10.214384    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c10371e-56e7-42ec-9ef8-0802e15cb2ff-tmp-dir\") pod \"premium-w4-0ad7c5-kserve-7bbf78cd4-pm6wc\" (UID: \"5c10371e-56e7-42ec-9ef8-0802e15cb2ff\") " pod="e2e-models-e2e-worker-w4-0ad7c5/premium-w4-0ad7c5-kserve-7bbf78cd4-pm6wc"
Oct 30 22:18:10.214455 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:18:10.214446    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c10371e-56e7-42ec-9ef8-0802e15cb2ff-home\") pod \"premium-w4-0ad7c5-kserve-7bbf78cd4-pm6wc\" (UID: \"5c10371e-56e7-42ec-9ef8-0802e15cb2ff\") " pod="e2e-models-e2e-worker-w4-0ad7c5/premium-w4-0ad7c5-kserve-7bbf78cd4-pm6wc"
Oct 30 22:18:10.216517 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:18:10.216495    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c10371e-56e7-42ec-9ef8-0802e15cb2ff-dshm\") pod \"premium-w4-0ad7c5-kserve-7bbf78cd4-pm6wc\" (UID: \"5c10371e-56e7-42ec-9ef8-0802e15cb2ff\") " pod="e2e-models-e2e-worker-w4-0ad7c5/premium-w4-0ad7c5-kserve-7bbf78cd4-pm6wc"
Oct 30 22:18:10.216790 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:18:10.216771    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c10371e-56e7-42ec-9ef8-0802e15cb2ff-tls-certs\") pod \"premium-w4-0ad7c5-kserve-7bbf78cd4-pm6wc\" (UID: \"5c10371e-56e7-42ec-9ef8-0802e15cb2ff\") " pod="e2e-models-e2e-worker-w4-0ad7c5/premium-w4-0ad7c5-kserve-7bbf78cd4-pm6wc"
Oct 30 22:18:10.234012 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:18:10.233987    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9l6s\" (UniqueName: \"kubernetes.io/projected/5c10371e-56e7-42ec-9ef8-0802e15cb2ff-kube-api-access-m9l6s\") pod \"premium-w4-0ad7c5-kserve-7bbf78cd4-pm6wc\" (UID: \"5c10371e-56e7-42ec-9ef8-0802e15cb2ff\") " pod="e2e-models-e2e-worker-w4-0ad7c5/premium-w4-0ad7c5-kserve-7bbf78cd4-pm6wc"
Oct 30 22:18:10.315171 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:18:10.315105    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0ad7c5/premium-w4-0ad7c5-kserve-7bbf78cd4-pm6wc"
Oct 30 22:18:10.610468 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:18:10.610374    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-0ad7c5/premium-w4-0ad7c5-kserve-7bbf78cd4-pm6wc"]
Oct 30 22:18:10.615270 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:18:10.615245    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5c10371e_56e7_42ec_9ef8_0802e15cb2ff.slice/crio-59bb12aad27a2bfa27356c911bdaec3827b96839dd378796c78baf753488cd25 WatchSource:0}: Error finding container 59bb12aad27a2bfa27356c911bdaec3827b96839dd378796c78baf753488cd25: Status 404 returned error can't find the container with id 59bb12aad27a2bfa27356c911bdaec3827b96839dd378796c78baf753488cd25
Oct 30 22:18:10.801869 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:18:10.801840    4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-54114e/premium-w5-54114e-kserve-57f6747bcc-7wjrw"
Oct 30 22:18:10.813983 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:18:10.813955    4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-54114e/premium-w5-54114e-kserve-57f6747bcc-7wjrw"
Oct 30 22:18:11.359485 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:18:11.359451    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0ad7c5/premium-w4-0ad7c5-kserve-7bbf78cd4-pm6wc" event={"ID":"5c10371e-56e7-42ec-9ef8-0802e15cb2ff","Type":"ContainerStarted","Data":"637557cda41a31bc56bb9a152f4d424881f7f5106ddff5d13c95faafcd60856e"}
Oct 30 22:18:11.359485 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:18:11.359495    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0ad7c5/premium-w4-0ad7c5-kserve-7bbf78cd4-pm6wc" event={"ID":"5c10371e-56e7-42ec-9ef8-0802e15cb2ff","Type":"ContainerStarted","Data":"59bb12aad27a2bfa27356c911bdaec3827b96839dd378796c78baf753488cd25"}
Oct 30 22:18:11.370869 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:18:11.370848    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-54114e/premium-w5-54114e-kserve-57f6747bcc-7wjrw"
Oct 30 22:18:11.395224 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:18:11.395182    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-0ad7c5/premium-w4-0ad7c5-kserve-7bbf78cd4-pm6wc" podStartSLOduration=2.395170023 podStartE2EDuration="2.395170023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:18:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:18:11.393923584 +0000 UTC m=+675.319849808" watchObservedRunningTime="2026-10-30 22:18:11.395170023 +0000 UTC m=+675.321096228"
Oct 30 22:18:11.521734 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:18:11.521705    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6bz92"]
Oct 30 22:18:12.316302 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:18:12.316275    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0ad7c5/premium-w4-0ad7c5-kserve-7bbf78cd4-pm6wc"
Oct 30 22:18:18.200440 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:18:18.200408    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-012ba0/premium-w6-012ba0-kserve-658654dbdb-d2k27"]
Oct 30 22:18:18.208365 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:18:18.208338    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-012ba0/premium-w6-012ba0-kserve-658654dbdb-d2k27"
Oct 30 22:18:18.232967 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:18:18.232932    4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-012ba0\"/\"openshift-service-ca.crt\""
Oct 30 22:18:18.233169 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:18:18.232970    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-012ba0\"/\"premium-w6-012ba0-kserve-self-signed-certs\""
Oct 30 22:18:18.233169 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:18:18.232972    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-012ba0\"/\"default-dockercfg-2wmx7\""
Oct 30 22:18:18.233169 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:18:18.233085    4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-012ba0\"/\"kube-root-ca.crt\""
Oct 30 22:18:18.234836 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:18:18.234814    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-012ba0/premium-w6-012ba0-kserve-658654dbdb-d2k27"]
Oct 30 22:18:18.285751 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:18:18.285725    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/085847e5-4f50-419e-984f-b612b104ed90-tls-certs\") pod \"premium-w6-012ba0-kserve-658654dbdb-d2k27\" (UID: \"085847e5-4f50-419e-984f-b612b104ed90\") " pod="e2e-models-e2e-worker-w6-012ba0/premium-w6-012ba0-kserve-658654dbdb-d2k27"
Oct 30 22:18:18.285852 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:18:18.285759    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/085847e5-4f50-419e-984f-b612b104ed90-tmp-dir\") pod \"premium-w6-012ba0-kserve-658654dbdb-d2k27\" (UID: \"085847e5-4f50-419e-984f-b612b104ed90\") " pod="e2e-models-e2e-worker-w6-012ba0/premium-w6-012ba0-kserve-658654dbdb-d2k27"
Oct 30 22:18:18.285852 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:18:18.285785    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnclv\" (UniqueName: \"kubernetes.io/projected/085847e5-4f50-419e-984f-b612b104ed90-kube-api-access-nnclv\") pod \"premium-w6-012ba0-kserve-658654dbdb-d2k27\" (UID: \"085847e5-4f50-419e-984f-b612b104ed90\") " pod="e2e-models-e2e-worker-w6-012ba0/premium-w6-012ba0-kserve-658654dbdb-d2k27"
Oct 30 22:18:18.285957 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:18:18.285861    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/085847e5-4f50-419e-984f-b612b104ed90-home\") pod \"premium-w6-012ba0-kserve-658654dbdb-d2k27\" (UID: \"085847e5-4f50-419e-984f-b612b104ed90\") " pod="e2e-models-e2e-worker-w6-012ba0/premium-w6-012ba0-kserve-658654dbdb-d2k27"
Oct 30 22:18:18.286103 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:18:18.286083    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/085847e5-4f50-419e-984f-b612b104ed90-model-cache\") pod \"premium-w6-012ba0-kserve-658654dbdb-d2k27\" (UID: \"085847e5-4f50-419e-984f-b612b104ed90\") " pod="e2e-models-e2e-worker-w6-012ba0/premium-w6-012ba0-kserve-658654dbdb-d2k27"
Oct 30 22:18:18.286178 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:18:18.286117    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/085847e5-4f50-419e-984f-b612b104ed90-dshm\") pod \"premium-w6-012ba0-kserve-658654dbdb-d2k27\" (UID: \"085847e5-4f50-419e-984f-b612b104ed90\") " pod="e2e-models-e2e-worker-w6-012ba0/premium-w6-012ba0-kserve-658654dbdb-d2k27"
Oct 30 22:18:18.387532 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:18:18.387504    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/085847e5-4f50-419e-984f-b612b104ed90-model-cache\") pod \"premium-w6-012ba0-kserve-658654dbdb-d2k27\" (UID: \"085847e5-4f50-419e-984f-b612b104ed90\") " pod="e2e-models-e2e-worker-w6-012ba0/premium-w6-012ba0-kserve-658654dbdb-d2k27"
Oct 30 22:18:18.387694 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:18:18.387547    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/085847e5-4f50-419e-984f-b612b104ed90-dshm\") pod \"premium-w6-012ba0-kserve-658654dbdb-d2k27\" (UID: \"085847e5-4f50-419e-984f-b612b104ed90\") " pod="e2e-models-e2e-worker-w6-012ba0/premium-w6-012ba0-kserve-658654dbdb-d2k27"
Oct 30 22:18:18.387694 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:18:18.387618    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/085847e5-4f50-419e-984f-b612b104ed90-tls-certs\") pod \"premium-w6-012ba0-kserve-658654dbdb-d2k27\" (UID: \"085847e5-4f50-419e-984f-b612b104ed90\") " pod="e2e-models-e2e-worker-w6-012ba0/premium-w6-012ba0-kserve-658654dbdb-d2k27"
Oct 30 22:18:18.387694 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:18:18.387654    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/085847e5-4f50-419e-984f-b612b104ed90-tmp-dir\") pod \"premium-w6-012ba0-kserve-658654dbdb-d2k27\" (UID: \"085847e5-4f50-419e-984f-b612b104ed90\") " pod="e2e-models-e2e-worker-w6-012ba0/premium-w6-012ba0-kserve-658654dbdb-d2k27"
Oct 30 22:18:18.387694 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:18:18.387692    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnclv\" (UniqueName: \"kubernetes.io/projected/085847e5-4f50-419e-984f-b612b104ed90-kube-api-access-nnclv\") pod \"premium-w6-012ba0-kserve-658654dbdb-d2k27\" (UID: \"085847e5-4f50-419e-984f-b612b104ed90\") " pod="e2e-models-e2e-worker-w6-012ba0/premium-w6-012ba0-kserve-658654dbdb-d2k27"
Oct 30 22:18:18.387908 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:18:18.387737    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/085847e5-4f50-419e-984f-b612b104ed90-home\") pod \"premium-w6-012ba0-kserve-658654dbdb-d2k27\" (UID: \"085847e5-4f50-419e-984f-b612b104ed90\") " pod="e2e-models-e2e-worker-w6-012ba0/premium-w6-012ba0-kserve-658654dbdb-d2k27"
Oct 30 22:18:18.388015 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:18:18.387991    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/085847e5-4f50-419e-984f-b612b104ed90-model-cache\") pod \"premium-w6-012ba0-kserve-658654dbdb-d2k27\" (UID: \"085847e5-4f50-419e-984f-b612b104ed90\") " pod="e2e-models-e2e-worker-w6-012ba0/premium-w6-012ba0-kserve-658654dbdb-d2k27"
Oct 30 22:18:18.388220 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:18:18.388083    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/085847e5-4f50-419e-984f-b612b104ed90-home\") pod \"premium-w6-012ba0-kserve-658654dbdb-d2k27\" (UID: \"085847e5-4f50-419e-984f-b612b104ed90\") " pod="e2e-models-e2e-worker-w6-012ba0/premium-w6-012ba0-kserve-658654dbdb-d2k27"
Oct 30 22:18:18.388311 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:18:18.388173    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/085847e5-4f50-419e-984f-b612b104ed90-tmp-dir\") pod \"premium-w6-012ba0-kserve-658654dbdb-d2k27\" (UID: \"085847e5-4f50-419e-984f-b612b104ed90\") " pod="e2e-models-e2e-worker-w6-012ba0/premium-w6-012ba0-kserve-658654dbdb-d2k27"
Oct 30 22:18:18.390339 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:18:18.390304    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/085847e5-4f50-419e-984f-b612b104ed90-dshm\") pod \"premium-w6-012ba0-kserve-658654dbdb-d2k27\" (UID: \"085847e5-4f50-419e-984f-b612b104ed90\") " pod="e2e-models-e2e-worker-w6-012ba0/premium-w6-012ba0-kserve-658654dbdb-d2k27"
Oct 30 22:18:18.390339 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:18:18.390307    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/085847e5-4f50-419e-984f-b612b104ed90-tls-certs\") pod \"premium-w6-012ba0-kserve-658654dbdb-d2k27\" (UID: \"085847e5-4f50-419e-984f-b612b104ed90\") " pod="e2e-models-e2e-worker-w6-012ba0/premium-w6-012ba0-kserve-658654dbdb-d2k27"
Oct 30 22:18:18.405608 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:18:18.405584    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnclv\" (UniqueName: \"kubernetes.io/projected/085847e5-4f50-419e-984f-b612b104ed90-kube-api-access-nnclv\") pod \"premium-w6-012ba0-kserve-658654dbdb-d2k27\" (UID: \"085847e5-4f50-419e-984f-b612b104ed90\") " pod="e2e-models-e2e-worker-w6-012ba0/premium-w6-012ba0-kserve-658654dbdb-d2k27"
Oct 30 22:18:18.523368 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:18:18.523281    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-012ba0/premium-w6-012ba0-kserve-658654dbdb-d2k27"
Oct 30 22:18:18.832817 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:18:18.832743    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-012ba0/premium-w6-012ba0-kserve-658654dbdb-d2k27"]
Oct 30 22:18:18.836818 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:18:18.836791    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod085847e5_4f50_419e_984f_b612b104ed90.slice/crio-ed317ea5f2fa3157b21723015a25614a78865bea0b31787b991fefe5ca36cdd2 WatchSource:0}: Error finding container ed317ea5f2fa3157b21723015a25614a78865bea0b31787b991fefe5ca36cdd2: Status 404 returned error can't find the container with id ed317ea5f2fa3157b21723015a25614a78865bea0b31787b991fefe5ca36cdd2
Oct 30 22:18:19.403149 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:18:19.403115    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-012ba0/premium-w6-012ba0-kserve-658654dbdb-d2k27" event={"ID":"085847e5-4f50-419e-984f-b612b104ed90","Type":"ContainerStarted","Data":"e546755e29e463e07191b8d4ffe3ca38d37c8980123729aa9b5621eb4939de7f"}
Oct 30 22:18:19.403610 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:18:19.403584    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-012ba0/premium-w6-012ba0-kserve-658654dbdb-d2k27" event={"ID":"085847e5-4f50-419e-984f-b612b104ed90","Type":"ContainerStarted","Data":"ed317ea5f2fa3157b21723015a25614a78865bea0b31787b991fefe5ca36cdd2"}
Oct 30 22:18:19.442318 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:18:19.442276    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-012ba0/premium-w6-012ba0-kserve-658654dbdb-d2k27" podStartSLOduration=1.44225936 podStartE2EDuration="1.44225936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:18:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:18:19.440755468 +0000 UTC m=+683.366681677" watchObservedRunningTime="2026-10-30 22:18:19.44225936 +0000 UTC m=+683.368185566"
Oct 30 22:18:19.524335 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:18:19.524309    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-012ba0/premium-w6-012ba0-kserve-658654dbdb-d2k27"
Oct 30 22:18:20.315514 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:18:20.315482    4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-0ad7c5/premium-w4-0ad7c5-kserve-7bbf78cd4-pm6wc"
Oct 30 22:18:20.334564 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:18:20.334541    4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-0ad7c5/premium-w4-0ad7c5-kserve-7bbf78cd4-pm6wc"
Oct 30 22:18:20.419558 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:18:20.419534    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0ad7c5/premium-w4-0ad7c5-kserve-7bbf78cd4-pm6wc"
Oct 30 22:18:26.265671 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:18:26.265637    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 22:18:27.618603 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:18:27.618573    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6bz92"]
Oct 30 22:18:28.523952 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:18:28.523920    4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-012ba0/premium-w6-012ba0-kserve-658654dbdb-d2k27"
Oct 30 22:18:28.536164 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:18:28.536130    4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-012ba0/premium-w6-012ba0-kserve-658654dbdb-d2k27"
Oct 30 22:18:29.471522 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:18:29.471489    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-012ba0/premium-w6-012ba0-kserve-658654dbdb-d2k27"
Oct 30 22:18:56.268485 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:18:56.268451    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 22:19:01.813628 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:01.813592    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6bz92"]
Oct 30 22:19:14.830622 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:14.830582    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be124eb7-dttc7"]
Oct 30 22:19:14.834376 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:14.834359    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be124eb7-dttc7"
Oct 30 22:19:14.847126 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:14.847106    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:19:14.857270 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:14.857253    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 22:19:14.859378 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:14.859356    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be124eb7-dttc7"]
Oct 30 22:19:14.907491 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:14.907466    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e886f4d-e60f-4668-b3b6-62baa7f0b185-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be124eb7-dttc7\" (UID: \"0e886f4d-e60f-4668-b3b6-62baa7f0b185\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be124eb7-dttc7"
Oct 30 22:19:14.907587 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:14.907503    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9j9t\" (UniqueName: \"kubernetes.io/projected/0e886f4d-e60f-4668-b3b6-62baa7f0b185-kube-api-access-j9j9t\") pod \"maas-api-revoke-sub-be124eb7-dttc7\" (UID: \"0e886f4d-e60f-4668-b3b6-62baa7f0b185\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be124eb7-dttc7"
Oct 30 22:19:14.907628 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:14.907584    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e886f4d-e60f-4668-b3b6-62baa7f0b185-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be124eb7-dttc7\" (UID: \"0e886f4d-e60f-4668-b3b6-62baa7f0b185\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be124eb7-dttc7"
Oct 30 22:19:15.008188 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:15.008162    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e886f4d-e60f-4668-b3b6-62baa7f0b185-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be124eb7-dttc7\" (UID: \"0e886f4d-e60f-4668-b3b6-62baa7f0b185\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be124eb7-dttc7"
Oct 30 22:19:15.008317 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:15.008205    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9j9t\" (UniqueName: \"kubernetes.io/projected/0e886f4d-e60f-4668-b3b6-62baa7f0b185-kube-api-access-j9j9t\") pod \"maas-api-revoke-sub-be124eb7-dttc7\" (UID: \"0e886f4d-e60f-4668-b3b6-62baa7f0b185\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be124eb7-dttc7"
Oct 30 22:19:15.008317 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:15.008250    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e886f4d-e60f-4668-b3b6-62baa7f0b185-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be124eb7-dttc7\" (UID: \"0e886f4d-e60f-4668-b3b6-62baa7f0b185\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be124eb7-dttc7"
Oct 30 22:19:15.008922 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:15.008897    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e886f4d-e60f-4668-b3b6-62baa7f0b185-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be124eb7-dttc7\" (UID: \"0e886f4d-e60f-4668-b3b6-62baa7f0b185\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be124eb7-dttc7"
Oct 30 22:19:15.014250 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:15.014229    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e886f4d-e60f-4668-b3b6-62baa7f0b185-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be124eb7-dttc7\" (UID: \"0e886f4d-e60f-4668-b3b6-62baa7f0b185\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be124eb7-dttc7"
Oct 30 22:19:15.026457 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:15.026397    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9j9t\" (UniqueName: \"kubernetes.io/projected/0e886f4d-e60f-4668-b3b6-62baa7f0b185-kube-api-access-j9j9t\") pod \"maas-api-revoke-sub-be124eb7-dttc7\" (UID: \"0e886f4d-e60f-4668-b3b6-62baa7f0b185\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be124eb7-dttc7"
Oct 30 22:19:15.147050 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:15.147023    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be124eb7-dttc7"
Oct 30 22:19:15.442406 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:15.442325    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be124eb7-dttc7"]
Oct 30 22:19:15.446580 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:19:15.446554    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e886f4d_e60f_4668_b3b6_62baa7f0b185.slice/crio-d56aeabb2e9fd02e05355b916bd95f309f8d58d4cfe2a1ec10a1b7500f257439 WatchSource:0}: Error finding container d56aeabb2e9fd02e05355b916bd95f309f8d58d4cfe2a1ec10a1b7500f257439: Status 404 returned error can't find the container with id d56aeabb2e9fd02e05355b916bd95f309f8d58d4cfe2a1ec10a1b7500f257439
Oct 30 22:19:15.680892 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:15.680856    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be124eb7-dttc7" event={"ID":"0e886f4d-e60f-4668-b3b6-62baa7f0b185","Type":"ContainerStarted","Data":"d56aeabb2e9fd02e05355b916bd95f309f8d58d4cfe2a1ec10a1b7500f257439"}
Oct 30 22:19:18.703422 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:18.703381    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be124eb7-dttc7" event={"ID":"0e886f4d-e60f-4668-b3b6-62baa7f0b185","Type":"ContainerStarted","Data":"ff2c0ab783168598f4f1c21d97e72667f97a385fca7c994c9f8b96dd3027f8db"}
Oct 30 22:19:19.713470 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:19.713425    4477 generic.go:356] "Generic (PLEG): container finished" podID="0e886f4d-e60f-4668-b3b6-62baa7f0b185" containerID="ff2c0ab783168598f4f1c21d97e72667f97a385fca7c994c9f8b96dd3027f8db" exitCode=0
Oct 30 22:19:19.713967 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:19.713564    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be124eb7-dttc7" event={"ID":"0e886f4d-e60f-4668-b3b6-62baa7f0b185","Type":"ContainerDied","Data":"ff2c0ab783168598f4f1c21d97e72667f97a385fca7c994c9f8b96dd3027f8db"}
Oct 30 22:19:19.918007 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:19.917977    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be124eb7-dttc7"
Oct 30 22:19:20.060350 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:20.060276    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e886f4d-e60f-4668-b3b6-62baa7f0b185-kube-api-access-j9j9t\" (UniqueName: \"kubernetes.io/projected/0e886f4d-e60f-4668-b3b6-62baa7f0b185-kube-api-access-j9j9t\") pod \"0e886f4d-e60f-4668-b3b6-62baa7f0b185\" (UID: \"0e886f4d-e60f-4668-b3b6-62baa7f0b185\") "
Oct 30 22:19:20.060350 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:20.060313    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e886f4d-e60f-4668-b3b6-62baa7f0b185-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e886f4d-e60f-4668-b3b6-62baa7f0b185-maas-api-xks-ca\") pod \"0e886f4d-e60f-4668-b3b6-62baa7f0b185\" (UID: \"0e886f4d-e60f-4668-b3b6-62baa7f0b185\") "
Oct 30 22:19:20.060537 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:20.060403    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e886f4d-e60f-4668-b3b6-62baa7f0b185-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e886f4d-e60f-4668-b3b6-62baa7f0b185-maas-api-openshift-ca\") pod \"0e886f4d-e60f-4668-b3b6-62baa7f0b185\" (UID: \"0e886f4d-e60f-4668-b3b6-62baa7f0b185\") "
Oct 30 22:19:20.060765 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:20.060736    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e886f4d-e60f-4668-b3b6-62baa7f0b185-maas-api-openshift-ca" pod "0e886f4d-e60f-4668-b3b6-62baa7f0b185" (UID: "0e886f4d-e60f-4668-b3b6-62baa7f0b185"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:19:20.060920 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:20.060905    4477 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e886f4d-e60f-4668-b3b6-62baa7f0b185-maas-api-openshift-ca\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:19:20.062509 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:20.062479    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e886f4d-e60f-4668-b3b6-62baa7f0b185-maas-api-xks-ca" pod "0e886f4d-e60f-4668-b3b6-62baa7f0b185" (UID: "0e886f4d-e60f-4668-b3b6-62baa7f0b185"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:19:20.062604 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:20.062482    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e886f4d-e60f-4668-b3b6-62baa7f0b185-kube-api-access-j9j9t" pod "0e886f4d-e60f-4668-b3b6-62baa7f0b185" (UID: "0e886f4d-e60f-4668-b3b6-62baa7f0b185"). InnerVolumeSpecName "kube-api-access-j9j9t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:19:20.161581 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:20.161559    4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9j9t\" (UniqueName: \"kubernetes.io/projected/0e886f4d-e60f-4668-b3b6-62baa7f0b185-kube-api-access-j9j9t\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:19:20.161581 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:20.161583    4477 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e886f4d-e60f-4668-b3b6-62baa7f0b185-maas-api-xks-ca\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:19:20.723007 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:20.722974    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be124eb7-dttc7"
Oct 30 22:19:20.723536 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:20.722970    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be124eb7-dttc7" event={"ID":"0e886f4d-e60f-4668-b3b6-62baa7f0b185","Type":"ContainerDied","Data":"d56aeabb2e9fd02e05355b916bd95f309f8d58d4cfe2a1ec10a1b7500f257439"}
Oct 30 22:19:20.723536 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:20.723133    4477 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d56aeabb2e9fd02e05355b916bd95f309f8d58d4cfe2a1ec10a1b7500f257439"
Oct 30 22:19:26.265265 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:19:26.265232    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 22:19:29.570939 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:29.570902    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc038a41-bsw8z"]
Oct 30 22:19:29.571742 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:29.571711    4477 cpu_manager.go:414] "containerMap: removing container" podUID="0e886f4d-e60f-4668-b3b6-62baa7f0b185" containerName="revoke-keys"
Oct 30 22:19:29.571742 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:29.571739    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e886f4d-e60f-4668-b3b6-62baa7f0b185" containerName="revoke-keys"
Oct 30 22:19:29.571921 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:29.571846    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e886f4d-e60f-4668-b3b6-62baa7f0b185" containerName="revoke-keys"
Oct 30 22:19:29.575277 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:29.575257    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc038a41-bsw8z"
Oct 30 22:19:29.589672 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:29.589645    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:19:29.600246 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:29.600226    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 22:19:29.602808 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:29.602786    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc038a41-bsw8z"]
Oct 30 22:19:29.647651 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:29.647629    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n56tq\" (UniqueName: \"kubernetes.io/projected/aa2d3862-6aa7-4f8b-be74-f2f48c2c520a-kube-api-access-n56tq\") pod \"maas-api-revoke-sub-dc038a41-bsw8z\" (UID: \"aa2d3862-6aa7-4f8b-be74-f2f48c2c520a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc038a41-bsw8z"
Oct 30 22:19:29.647724 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:29.647672    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa2d3862-6aa7-4f8b-be74-f2f48c2c520a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc038a41-bsw8z\" (UID: \"aa2d3862-6aa7-4f8b-be74-f2f48c2c520a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc038a41-bsw8z"
Oct 30 22:19:29.647797 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:29.647784    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa2d3862-6aa7-4f8b-be74-f2f48c2c520a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc038a41-bsw8z\" (UID: \"aa2d3862-6aa7-4f8b-be74-f2f48c2c520a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc038a41-bsw8z"
Oct 30 22:19:29.749019 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:29.748997    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n56tq\" (UniqueName: \"kubernetes.io/projected/aa2d3862-6aa7-4f8b-be74-f2f48c2c520a-kube-api-access-n56tq\") pod \"maas-api-revoke-sub-dc038a41-bsw8z\" (UID: \"aa2d3862-6aa7-4f8b-be74-f2f48c2c520a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc038a41-bsw8z"
Oct 30 22:19:29.749126 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:29.749060    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa2d3862-6aa7-4f8b-be74-f2f48c2c520a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc038a41-bsw8z\" (UID: \"aa2d3862-6aa7-4f8b-be74-f2f48c2c520a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc038a41-bsw8z"
Oct 30 22:19:29.749181 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:29.749164    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa2d3862-6aa7-4f8b-be74-f2f48c2c520a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc038a41-bsw8z\" (UID: \"aa2d3862-6aa7-4f8b-be74-f2f48c2c520a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc038a41-bsw8z"
Oct 30 22:19:29.749773 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:29.749752    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa2d3862-6aa7-4f8b-be74-f2f48c2c520a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc038a41-bsw8z\" (UID: \"aa2d3862-6aa7-4f8b-be74-f2f48c2c520a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc038a41-bsw8z"
Oct 30 22:19:29.751292 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:29.751268    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa2d3862-6aa7-4f8b-be74-f2f48c2c520a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc038a41-bsw8z\" (UID: \"aa2d3862-6aa7-4f8b-be74-f2f48c2c520a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc038a41-bsw8z"
Oct 30 22:19:29.766169 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:29.766143    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n56tq\" (UniqueName: \"kubernetes.io/projected/aa2d3862-6aa7-4f8b-be74-f2f48c2c520a-kube-api-access-n56tq\") pod \"maas-api-revoke-sub-dc038a41-bsw8z\" (UID: \"aa2d3862-6aa7-4f8b-be74-f2f48c2c520a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc038a41-bsw8z"
Oct 30 22:19:29.888843 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:29.888822    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc038a41-bsw8z"
Oct 30 22:19:30.183843 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:30.183719    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc038a41-bsw8z"]
Oct 30 22:19:30.189272 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:19:30.189245    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa2d3862_6aa7_4f8b_be74_f2f48c2c520a.slice/crio-df1ce7703d819c771217490c03cf734bf4e7111a2abfbd57cfc2550b756f0b0a WatchSource:0}: Error finding container df1ce7703d819c771217490c03cf734bf4e7111a2abfbd57cfc2550b756f0b0a: Status 404 returned error can't find the container with id df1ce7703d819c771217490c03cf734bf4e7111a2abfbd57cfc2550b756f0b0a
Oct 30 22:19:30.773878 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:30.773843    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc038a41-bsw8z" event={"ID":"aa2d3862-6aa7-4f8b-be74-f2f48c2c520a","Type":"ContainerStarted","Data":"e161d97fac4599225c9b806cf2745ea9d4746771f5acd0226645341cdfa3b42f"}
Oct 30 22:19:30.774275 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:30.773898    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc038a41-bsw8z" event={"ID":"aa2d3862-6aa7-4f8b-be74-f2f48c2c520a","Type":"ContainerStarted","Data":"df1ce7703d819c771217490c03cf734bf4e7111a2abfbd57cfc2550b756f0b0a"}
Oct 30 22:19:30.819815 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:30.819765    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc038a41-bsw8z" podStartSLOduration=1.819753211 podStartE2EDuration="1.819753211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:19:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:19:30.819018122 +0000 UTC m=+754.744944328" watchObservedRunningTime="2026-10-30 22:19:30.819753211 +0000 UTC m=+754.745679416"
Oct 30 22:19:31.780138 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:31.780098    4477 generic.go:356] "Generic (PLEG): container finished" podID="aa2d3862-6aa7-4f8b-be74-f2f48c2c520a" containerID="e161d97fac4599225c9b806cf2745ea9d4746771f5acd0226645341cdfa3b42f" exitCode=0
Oct 30 22:19:31.780592 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:31.780162    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc038a41-bsw8z" event={"ID":"aa2d3862-6aa7-4f8b-be74-f2f48c2c520a","Type":"ContainerDied","Data":"e161d97fac4599225c9b806cf2745ea9d4746771f5acd0226645341cdfa3b42f"}
Oct 30 22:19:32.917545 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:32.917518    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc038a41-bsw8z"
Oct 30 22:19:33.080801 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:33.080724    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa2d3862-6aa7-4f8b-be74-f2f48c2c520a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa2d3862-6aa7-4f8b-be74-f2f48c2c520a-maas-api-xks-ca\") pod \"aa2d3862-6aa7-4f8b-be74-f2f48c2c520a\" (UID: \"aa2d3862-6aa7-4f8b-be74-f2f48c2c520a\") "
Oct 30 22:19:33.080801 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:33.080768    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa2d3862-6aa7-4f8b-be74-f2f48c2c520a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa2d3862-6aa7-4f8b-be74-f2f48c2c520a-maas-api-openshift-ca\") pod \"aa2d3862-6aa7-4f8b-be74-f2f48c2c520a\" (UID: \"aa2d3862-6aa7-4f8b-be74-f2f48c2c520a\") "
Oct 30 22:19:33.080983 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:33.080868    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa2d3862-6aa7-4f8b-be74-f2f48c2c520a-kube-api-access-n56tq\" (UniqueName: \"kubernetes.io/projected/aa2d3862-6aa7-4f8b-be74-f2f48c2c520a-kube-api-access-n56tq\") pod \"aa2d3862-6aa7-4f8b-be74-f2f48c2c520a\" (UID: \"aa2d3862-6aa7-4f8b-be74-f2f48c2c520a\") "
Oct 30 22:19:33.081195 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:33.081171    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa2d3862-6aa7-4f8b-be74-f2f48c2c520a-maas-api-openshift-ca" pod "aa2d3862-6aa7-4f8b-be74-f2f48c2c520a" (UID: "aa2d3862-6aa7-4f8b-be74-f2f48c2c520a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:19:33.082860 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:33.082841    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa2d3862-6aa7-4f8b-be74-f2f48c2c520a-maas-api-xks-ca" pod "aa2d3862-6aa7-4f8b-be74-f2f48c2c520a" (UID: "aa2d3862-6aa7-4f8b-be74-f2f48c2c520a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:19:33.082939 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:33.082924    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa2d3862-6aa7-4f8b-be74-f2f48c2c520a-kube-api-access-n56tq" pod "aa2d3862-6aa7-4f8b-be74-f2f48c2c520a" (UID: "aa2d3862-6aa7-4f8b-be74-f2f48c2c520a"). InnerVolumeSpecName "kube-api-access-n56tq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:19:33.182290 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:33.182253    4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n56tq\" (UniqueName: \"kubernetes.io/projected/aa2d3862-6aa7-4f8b-be74-f2f48c2c520a-kube-api-access-n56tq\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:19:33.182290 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:33.182283    4477 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa2d3862-6aa7-4f8b-be74-f2f48c2c520a-maas-api-xks-ca\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:19:33.182290 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:33.182292    4477 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa2d3862-6aa7-4f8b-be74-f2f48c2c520a-maas-api-openshift-ca\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:19:33.791716 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:33.791675    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc038a41-bsw8z" event={"ID":"aa2d3862-6aa7-4f8b-be74-f2f48c2c520a","Type":"ContainerDied","Data":"df1ce7703d819c771217490c03cf734bf4e7111a2abfbd57cfc2550b756f0b0a"}
Oct 30 22:19:33.791879 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:33.791735    4477 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df1ce7703d819c771217490c03cf734bf4e7111a2abfbd57cfc2550b756f0b0a"
Oct 30 22:19:33.791879 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:33.791696    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc038a41-bsw8z"
Oct 30 22:19:34.798894 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:34.798862    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6bz92"]
Oct 30 22:19:36.473763 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:36.473734    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hv7z4"]
Oct 30 22:19:36.474433 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:36.474412    4477 cpu_manager.go:414] "containerMap: removing container" podUID="aa2d3862-6aa7-4f8b-be74-f2f48c2c520a" containerName="revoke-keys"
Oct 30 22:19:36.474433 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:36.474433    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa2d3862-6aa7-4f8b-be74-f2f48c2c520a" containerName="revoke-keys"
Oct 30 22:19:36.474697 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:36.474581    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa2d3862-6aa7-4f8b-be74-f2f48c2c520a" containerName="revoke-keys"
Oct 30 22:19:36.478963 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:36.478941    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hv7z4"
Oct 30 22:19:36.509077 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:36.506268    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hv7z4"]
Oct 30 22:19:36.615132 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:36.615097    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-hv7z4\" (UID: \"8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hv7z4"
Oct 30 22:19:36.615278 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:36.615147    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-hv7z4\" (UID: \"8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hv7z4"
Oct 30 22:19:36.615339 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:36.615273    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-hv7z4\" (UID: \"8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hv7z4"
Oct 30 22:19:36.615415 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:36.615395    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-hv7z4\" (UID: \"8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hv7z4"
Oct 30 22:19:36.615475 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:36.615435    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-hv7z4\" (UID: \"8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hv7z4"
Oct 30 22:19:36.615516 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:36.615504    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-hv7z4\" (UID: \"8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hv7z4"
Oct 30 22:19:36.615558 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:36.615533    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-hv7z4\" (UID: \"8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hv7z4"
Oct 30 22:19:36.615633 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:36.615616    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5d2fv\" (UniqueName: \"kubernetes.io/projected/8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d-kube-api-access-5d2fv\") pod \"maas-default-gateway-openshift-default-6db586dd6c-hv7z4\" (UID: \"8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hv7z4"
Oct 30 22:19:36.615695 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:36.615654    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-hv7z4\" (UID: \"8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hv7z4"
Oct 30 22:19:36.717121 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:36.717091    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-hv7z4\" (UID: \"8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hv7z4"
Oct 30 22:19:36.717263 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:36.717131    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-hv7z4\" (UID: \"8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hv7z4"
Oct 30 22:19:36.717263 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:36.717158    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-hv7z4\" (UID: \"8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hv7z4"
Oct 30 22:19:36.717263 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:36.717204    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-hv7z4\" (UID: \"8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hv7z4"
Oct 30 22:19:36.717263 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:36.717229    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-hv7z4\" (UID: \"8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hv7z4"
Oct 30 22:19:36.717263 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:36.717258    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-hv7z4\" (UID: \"8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hv7z4"
Oct 30 22:19:36.717527 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:36.717283    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-hv7z4\" (UID: \"8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hv7z4"
Oct 30 22:19:36.717527 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:36.717318    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5d2fv\" (UniqueName: \"kubernetes.io/projected/8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d-kube-api-access-5d2fv\") pod \"maas-default-gateway-openshift-default-6db586dd6c-hv7z4\" (UID: \"8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hv7z4"
Oct 30 22:19:36.717527 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:36.717350    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-hv7z4\" (UID: \"8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hv7z4"
Oct 30 22:19:36.717527 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:36.717516    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-hv7z4\" (UID: \"8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hv7z4"
Oct 30 22:19:36.717720 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:36.717572    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-hv7z4\" (UID: \"8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hv7z4"
Oct 30 22:19:36.718195 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:36.718151    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-hv7z4\" (UID: \"8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hv7z4"
Oct 30 22:19:36.718315 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:36.718300    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-hv7z4\" (UID: \"8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hv7z4"
Oct 30 22:19:36.718573 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:36.718547    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-hv7z4\" (UID: \"8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hv7z4"
Oct 30 22:19:36.719813 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:36.719786    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-hv7z4\" (UID: \"8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hv7z4"
Oct 30 22:19:36.720297 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:36.720275    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-hv7z4\" (UID: \"8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hv7z4"
Oct 30 22:19:36.735023 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:36.734963    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-hv7z4\" (UID: \"8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hv7z4"
Oct 30 22:19:36.735250 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:36.735227    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5d2fv\" (UniqueName: \"kubernetes.io/projected/8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d-kube-api-access-5d2fv\") pod \"maas-default-gateway-openshift-default-6db586dd6c-hv7z4\" (UID: \"8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hv7z4"
Oct 30 22:19:36.795287 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:36.795258    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hv7z4"
Oct 30 22:19:37.117290 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:37.117181    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hv7z4"]
Oct 30 22:19:37.128067 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:37.127802    4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:19:37.128067 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:37.127887    4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:19:37.128067 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:37.127933    4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:19:37.833064 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:37.830801    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hv7z4" event={"ID":"8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d","Type":"ContainerStarted","Data":"f35c87e53b11d3dcb55acab8d5a6a8c385a6c29e5972da230e3011295ba5cbb8"}
Oct 30 22:19:37.833064 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:37.830909    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hv7z4" event={"ID":"8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d","Type":"ContainerStarted","Data":"561c6af02147db61debe1c956b38b8c66083cf009784484265105d748428bae5"}
Oct 30 22:19:37.895001 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:37.894601    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hv7z4" podStartSLOduration=1.894582637 podStartE2EDuration="1.894582637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:19:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:19:37.879063439 +0000 UTC m=+761.804989649" watchObservedRunningTime="2026-10-30 22:19:37.894582637 +0000 UTC m=+761.820508843"
Oct 30 22:19:38.796484 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:38.796450    4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hv7z4"
Oct 30 22:19:38.798256 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:38.798229    4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-hv7z4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Oct 30 22:19:38.798402 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:38.798299    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hv7z4" podUID="8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Oct 30 22:19:38.903211 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:38.903080    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-d52c59/distinct-w3-d52c59-kserve-ffb5f5b96-l58xl"]
Oct 30 22:19:38.907814 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:38.907785    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d52c59/distinct-w3-d52c59-kserve-ffb5f5b96-l58xl"
Oct 30 22:19:38.921268 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:38.921243    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-d52c59/distinct-w3-d52c59-kserve-ffb5f5b96-l58xl"]
Oct 30 22:19:38.928462 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:38.928433    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-d52c59\"/\"distinct-w3-d52c59-kserve-self-signed-certs\""
Oct 30 22:19:38.928601 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:38.928559    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-d52c59\"/\"default-dockercfg-pq7cw\""
Oct 30 22:19:38.930716 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:38.930685    4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-d52c59\"/\"openshift-service-ca.crt\""
Oct 30 22:19:38.930825 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:38.930792    4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-d52c59\"/\"kube-root-ca.crt\""
Oct 30 22:19:39.049364 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:39.049275    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7b1483b-2631-4aa2-a888-3096761d5494-home\") pod \"distinct-w3-d52c59-kserve-ffb5f5b96-l58xl\" (UID: \"c7b1483b-2631-4aa2-a888-3096761d5494\") " pod="e2e-models-e2e-worker-w3-d52c59/distinct-w3-d52c59-kserve-ffb5f5b96-l58xl"
Oct 30 22:19:39.049364 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:39.049353    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsc7p\" (UniqueName: \"kubernetes.io/projected/c7b1483b-2631-4aa2-a888-3096761d5494-kube-api-access-bsc7p\") pod \"distinct-w3-d52c59-kserve-ffb5f5b96-l58xl\" (UID: \"c7b1483b-2631-4aa2-a888-3096761d5494\") " pod="e2e-models-e2e-worker-w3-d52c59/distinct-w3-d52c59-kserve-ffb5f5b96-l58xl"
Oct 30 22:19:39.049579 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:39.049456    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7b1483b-2631-4aa2-a888-3096761d5494-model-cache\") pod \"distinct-w3-d52c59-kserve-ffb5f5b96-l58xl\" (UID: \"c7b1483b-2631-4aa2-a888-3096761d5494\") " pod="e2e-models-e2e-worker-w3-d52c59/distinct-w3-d52c59-kserve-ffb5f5b96-l58xl"
Oct 30 22:19:39.049579 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:39.049527    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7b1483b-2631-4aa2-a888-3096761d5494-dshm\") pod \"distinct-w3-d52c59-kserve-ffb5f5b96-l58xl\" (UID: \"c7b1483b-2631-4aa2-a888-3096761d5494\") " pod="e2e-models-e2e-worker-w3-d52c59/distinct-w3-d52c59-kserve-ffb5f5b96-l58xl"
Oct 30 22:19:39.049579 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:39.049572    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7b1483b-2631-4aa2-a888-3096761d5494-tmp-dir\") pod \"distinct-w3-d52c59-kserve-ffb5f5b96-l58xl\" (UID: \"c7b1483b-2631-4aa2-a888-3096761d5494\") " pod="e2e-models-e2e-worker-w3-d52c59/distinct-w3-d52c59-kserve-ffb5f5b96-l58xl"
Oct 30 22:19:39.049726 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:39.049636    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7b1483b-2631-4aa2-a888-3096761d5494-tls-certs\") pod \"distinct-w3-d52c59-kserve-ffb5f5b96-l58xl\" (UID: \"c7b1483b-2631-4aa2-a888-3096761d5494\") " pod="e2e-models-e2e-worker-w3-d52c59/distinct-w3-d52c59-kserve-ffb5f5b96-l58xl"
Oct 30 22:19:39.150637 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:39.150604    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7b1483b-2631-4aa2-a888-3096761d5494-home\") pod \"distinct-w3-d52c59-kserve-ffb5f5b96-l58xl\" (UID: \"c7b1483b-2631-4aa2-a888-3096761d5494\") " pod="e2e-models-e2e-worker-w3-d52c59/distinct-w3-d52c59-kserve-ffb5f5b96-l58xl"
Oct 30 22:19:39.150830 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:39.150652    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsc7p\" (UniqueName: \"kubernetes.io/projected/c7b1483b-2631-4aa2-a888-3096761d5494-kube-api-access-bsc7p\") pod \"distinct-w3-d52c59-kserve-ffb5f5b96-l58xl\" (UID: \"c7b1483b-2631-4aa2-a888-3096761d5494\") " pod="e2e-models-e2e-worker-w3-d52c59/distinct-w3-d52c59-kserve-ffb5f5b96-l58xl"
Oct 30 22:19:39.150830 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:39.150689    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7b1483b-2631-4aa2-a888-3096761d5494-model-cache\") pod \"distinct-w3-d52c59-kserve-ffb5f5b96-l58xl\" (UID: \"c7b1483b-2631-4aa2-a888-3096761d5494\") " pod="e2e-models-e2e-worker-w3-d52c59/distinct-w3-d52c59-kserve-ffb5f5b96-l58xl"
Oct 30 22:19:39.150830 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:39.150707    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7b1483b-2631-4aa2-a888-3096761d5494-dshm\") pod \"distinct-w3-d52c59-kserve-ffb5f5b96-l58xl\" (UID: \"c7b1483b-2631-4aa2-a888-3096761d5494\") " pod="e2e-models-e2e-worker-w3-d52c59/distinct-w3-d52c59-kserve-ffb5f5b96-l58xl"
Oct 30 22:19:39.150830 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:39.150727    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7b1483b-2631-4aa2-a888-3096761d5494-tmp-dir\") pod \"distinct-w3-d52c59-kserve-ffb5f5b96-l58xl\" (UID: \"c7b1483b-2631-4aa2-a888-3096761d5494\") " pod="e2e-models-e2e-worker-w3-d52c59/distinct-w3-d52c59-kserve-ffb5f5b96-l58xl"
Oct 30 22:19:39.150830 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:39.150804    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7b1483b-2631-4aa2-a888-3096761d5494-tls-certs\") pod \"distinct-w3-d52c59-kserve-ffb5f5b96-l58xl\" (UID: \"c7b1483b-2631-4aa2-a888-3096761d5494\") " pod="e2e-models-e2e-worker-w3-d52c59/distinct-w3-d52c59-kserve-ffb5f5b96-l58xl"
Oct 30 22:19:39.151312 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:39.151280    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7b1483b-2631-4aa2-a888-3096761d5494-model-cache\") pod \"distinct-w3-d52c59-kserve-ffb5f5b96-l58xl\" (UID: \"c7b1483b-2631-4aa2-a888-3096761d5494\") " pod="e2e-models-e2e-worker-w3-d52c59/distinct-w3-d52c59-kserve-ffb5f5b96-l58xl"
Oct 30 22:19:39.151428 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:39.151336    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7b1483b-2631-4aa2-a888-3096761d5494-home\") pod \"distinct-w3-d52c59-kserve-ffb5f5b96-l58xl\" (UID: \"c7b1483b-2631-4aa2-a888-3096761d5494\") " pod="e2e-models-e2e-worker-w3-d52c59/distinct-w3-d52c59-kserve-ffb5f5b96-l58xl"
Oct 30 22:19:39.151428 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:39.151367    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7b1483b-2631-4aa2-a888-3096761d5494-tmp-dir\") pod \"distinct-w3-d52c59-kserve-ffb5f5b96-l58xl\" (UID: \"c7b1483b-2631-4aa2-a888-3096761d5494\") " pod="e2e-models-e2e-worker-w3-d52c59/distinct-w3-d52c59-kserve-ffb5f5b96-l58xl"
Oct 30 22:19:39.153156 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:39.153133    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7b1483b-2631-4aa2-a888-3096761d5494-dshm\") pod \"distinct-w3-d52c59-kserve-ffb5f5b96-l58xl\" (UID: \"c7b1483b-2631-4aa2-a888-3096761d5494\") " pod="e2e-models-e2e-worker-w3-d52c59/distinct-w3-d52c59-kserve-ffb5f5b96-l58xl"
Oct 30 22:19:39.153795 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:39.153774    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7b1483b-2631-4aa2-a888-3096761d5494-tls-certs\") pod \"distinct-w3-d52c59-kserve-ffb5f5b96-l58xl\" (UID: \"c7b1483b-2631-4aa2-a888-3096761d5494\") " pod="e2e-models-e2e-worker-w3-d52c59/distinct-w3-d52c59-kserve-ffb5f5b96-l58xl"
Oct 30 22:19:39.167933 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:39.167901    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsc7p\" (UniqueName: \"kubernetes.io/projected/c7b1483b-2631-4aa2-a888-3096761d5494-kube-api-access-bsc7p\") pod \"distinct-w3-d52c59-kserve-ffb5f5b96-l58xl\" (UID: \"c7b1483b-2631-4aa2-a888-3096761d5494\") " pod="e2e-models-e2e-worker-w3-d52c59/distinct-w3-d52c59-kserve-ffb5f5b96-l58xl"
Oct 30 22:19:39.227655 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:39.227622    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d52c59/distinct-w3-d52c59-kserve-ffb5f5b96-l58xl"
Oct 30 22:19:39.552163 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:39.552077    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-d52c59/distinct-w3-d52c59-kserve-ffb5f5b96-l58xl"]
Oct 30 22:19:39.556946 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:19:39.556889    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc7b1483b_2631_4aa2_a888_3096761d5494.slice/crio-5d2db8ec0feb870a7866ef8138804196aa348c182a4b7c8c1f702ee2e9208bda WatchSource:0}: Error finding container 5d2db8ec0feb870a7866ef8138804196aa348c182a4b7c8c1f702ee2e9208bda: Status 404 returned error can't find the container with id 5d2db8ec0feb870a7866ef8138804196aa348c182a4b7c8c1f702ee2e9208bda
Oct 30 22:19:39.800067 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:39.799317    4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-hv7z4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Oct 30 22:19:39.800067 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:39.799382    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hv7z4" podUID="8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Oct 30 22:19:39.842925 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:39.842844    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d52c59/distinct-w3-d52c59-kserve-ffb5f5b96-l58xl" event={"ID":"c7b1483b-2631-4aa2-a888-3096761d5494","Type":"ContainerStarted","Data":"a543823ac89b6d62bf8c1e49348e2349a689bbacde460605ed87092c88e9df96"}
Oct 30 22:19:39.842925 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:39.842896    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d52c59/distinct-w3-d52c59-kserve-ffb5f5b96-l58xl" event={"ID":"c7b1483b-2631-4aa2-a888-3096761d5494","Type":"ContainerStarted","Data":"5d2db8ec0feb870a7866ef8138804196aa348c182a4b7c8c1f702ee2e9208bda"}
Oct 30 22:19:40.228002 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:40.227973    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d52c59/distinct-w3-d52c59-kserve-ffb5f5b96-l58xl"
Oct 30 22:19:40.801064 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:40.800414    4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-hv7z4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Oct 30 22:19:40.801064 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:40.800474    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hv7z4" podUID="8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Oct 30 22:19:41.796585 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:41.796544    4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-hv7z4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Oct 30 22:19:41.797079 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:41.796602    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hv7z4" podUID="8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Oct 30 22:19:42.796591 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:42.796554    4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-hv7z4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Oct 30 22:19:42.796960 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:42.796628    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hv7z4" podUID="8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Oct 30 22:19:43.796035 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:43.795999    4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-hv7z4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Oct 30 22:19:43.796197 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:43.796081    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hv7z4" podUID="8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Oct 30 22:19:43.951457 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:43.951405    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-d52c59/distinct-w3-d52c59-kserve-ffb5f5b96-l58xl" podStartSLOduration=5.951391914 podStartE2EDuration="5.951391914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:19:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:19:39.884170696 +0000 UTC m=+763.810096919" watchObservedRunningTime="2026-10-30 22:19:43.951391914 +0000 UTC m=+767.877318138"
Oct 30 22:19:43.952570 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:43.952545    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d9b4df1-xrwbz"]
Oct 30 22:19:43.956360 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:43.956341    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d9b4df1-xrwbz"
Oct 30 22:19:43.979176 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:43.979155    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:19:43.979281 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:43.979243    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 22:19:43.982289 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:43.982266    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d9b4df1-xrwbz"]
Oct 30 22:19:44.020453 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:44.020428    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkv6k\" (UniqueName: \"kubernetes.io/projected/a7fd3273-aa8d-466e-a0e6-ec57f8723db6-kube-api-access-dkv6k\") pod \"maas-api-revoke-sub-2d9b4df1-xrwbz\" (UID: \"a7fd3273-aa8d-466e-a0e6-ec57f8723db6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d9b4df1-xrwbz"
Oct 30 22:19:44.020590 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:44.020490    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7fd3273-aa8d-466e-a0e6-ec57f8723db6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d9b4df1-xrwbz\" (UID: \"a7fd3273-aa8d-466e-a0e6-ec57f8723db6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d9b4df1-xrwbz"
Oct 30 22:19:44.020638 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:44.020585    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7fd3273-aa8d-466e-a0e6-ec57f8723db6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d9b4df1-xrwbz\" (UID: \"a7fd3273-aa8d-466e-a0e6-ec57f8723db6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d9b4df1-xrwbz"
Oct 30 22:19:44.121580 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:44.121499    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkv6k\" (UniqueName: \"kubernetes.io/projected/a7fd3273-aa8d-466e-a0e6-ec57f8723db6-kube-api-access-dkv6k\") pod \"maas-api-revoke-sub-2d9b4df1-xrwbz\" (UID: \"a7fd3273-aa8d-466e-a0e6-ec57f8723db6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d9b4df1-xrwbz"
Oct 30 22:19:44.121742 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:44.121584    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7fd3273-aa8d-466e-a0e6-ec57f8723db6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d9b4df1-xrwbz\" (UID: \"a7fd3273-aa8d-466e-a0e6-ec57f8723db6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d9b4df1-xrwbz"
Oct 30 22:19:44.121742 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:44.121640    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7fd3273-aa8d-466e-a0e6-ec57f8723db6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d9b4df1-xrwbz\" (UID: \"a7fd3273-aa8d-466e-a0e6-ec57f8723db6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d9b4df1-xrwbz"
Oct 30 22:19:44.122354 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:44.122323    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7fd3273-aa8d-466e-a0e6-ec57f8723db6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d9b4df1-xrwbz\" (UID: \"a7fd3273-aa8d-466e-a0e6-ec57f8723db6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d9b4df1-xrwbz"
Oct 30 22:19:44.123889 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:44.123867    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7fd3273-aa8d-466e-a0e6-ec57f8723db6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d9b4df1-xrwbz\" (UID: \"a7fd3273-aa8d-466e-a0e6-ec57f8723db6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d9b4df1-xrwbz"
Oct 30 22:19:44.138743 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:44.138718    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkv6k\" (UniqueName: \"kubernetes.io/projected/a7fd3273-aa8d-466e-a0e6-ec57f8723db6-kube-api-access-dkv6k\") pod \"maas-api-revoke-sub-2d9b4df1-xrwbz\" (UID: \"a7fd3273-aa8d-466e-a0e6-ec57f8723db6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d9b4df1-xrwbz"
Oct 30 22:19:44.270434 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:44.270401    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d9b4df1-xrwbz"
Oct 30 22:19:44.564551 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:44.564517    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d9b4df1-xrwbz"]
Oct 30 22:19:44.569182 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:19:44.569155    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7fd3273_aa8d_466e_a0e6_ec57f8723db6.slice/crio-88e99f2043c2b3cbe5ad958afd11f999cf66deb94095bd166bb8392cc51ce894 WatchSource:0}: Error finding container 88e99f2043c2b3cbe5ad958afd11f999cf66deb94095bd166bb8392cc51ce894: Status 404 returned error can't find the container with id 88e99f2043c2b3cbe5ad958afd11f999cf66deb94095bd166bb8392cc51ce894
Oct 30 22:19:44.796501 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:44.796476    4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-hv7z4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body=
Oct 30 22:19:44.796627 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:44.796528    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hv7z4" podUID="8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused"
Oct 30 22:19:44.870979 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:44.870920    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d9b4df1-xrwbz" event={"ID":"a7fd3273-aa8d-466e-a0e6-ec57f8723db6","Type":"ContainerStarted","Data":"88e99f2043c2b3cbe5ad958afd11f999cf66deb94095bd166bb8392cc51ce894"}
Oct 30 22:19:45.878723 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:45.878692    4477 generic.go:356] "Generic (PLEG): container finished" podID="a7fd3273-aa8d-466e-a0e6-ec57f8723db6" containerID="76ad4847df1bbc11233f262f75dceb46c3a898ebc0e6df2243c4b4045db47639" exitCode=0
Oct 30 22:19:45.879279 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:45.878777    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d9b4df1-xrwbz" event={"ID":"a7fd3273-aa8d-466e-a0e6-ec57f8723db6","Type":"ContainerDied","Data":"76ad4847df1bbc11233f262f75dceb46c3a898ebc0e6df2243c4b4045db47639"}
Oct 30 22:19:45.998250 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:45.998216    4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hv7z4"
Oct 30 22:19:45.998898 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:45.998752    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hv7z4"
Oct 30 22:19:45.999901 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:45.999878    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hv7z4"
Oct 30 22:19:47.022518 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:47.022493    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d9b4df1-xrwbz"
Oct 30 22:19:47.054742 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:47.054720    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7fd3273-aa8d-466e-a0e6-ec57f8723db6-kube-api-access-dkv6k\" (UniqueName: \"kubernetes.io/projected/a7fd3273-aa8d-466e-a0e6-ec57f8723db6-kube-api-access-dkv6k\") pod \"a7fd3273-aa8d-466e-a0e6-ec57f8723db6\" (UID: \"a7fd3273-aa8d-466e-a0e6-ec57f8723db6\") "
Oct 30 22:19:47.054854 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:47.054814    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7fd3273-aa8d-466e-a0e6-ec57f8723db6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7fd3273-aa8d-466e-a0e6-ec57f8723db6-maas-api-xks-ca\") pod \"a7fd3273-aa8d-466e-a0e6-ec57f8723db6\" (UID: \"a7fd3273-aa8d-466e-a0e6-ec57f8723db6\") "
Oct 30 22:19:47.054854 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:47.054847    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7fd3273-aa8d-466e-a0e6-ec57f8723db6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7fd3273-aa8d-466e-a0e6-ec57f8723db6-maas-api-openshift-ca\") pod \"a7fd3273-aa8d-466e-a0e6-ec57f8723db6\" (UID: \"a7fd3273-aa8d-466e-a0e6-ec57f8723db6\") "
Oct 30 22:19:47.055239 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:47.055212    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7fd3273-aa8d-466e-a0e6-ec57f8723db6-maas-api-openshift-ca" pod "a7fd3273-aa8d-466e-a0e6-ec57f8723db6" (UID: "a7fd3273-aa8d-466e-a0e6-ec57f8723db6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:19:47.055344 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:47.055298    4477 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7fd3273-aa8d-466e-a0e6-ec57f8723db6-maas-api-openshift-ca\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:19:47.056698 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:47.056671    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7fd3273-aa8d-466e-a0e6-ec57f8723db6-kube-api-access-dkv6k" pod "a7fd3273-aa8d-466e-a0e6-ec57f8723db6" (UID: "a7fd3273-aa8d-466e-a0e6-ec57f8723db6"). InnerVolumeSpecName "kube-api-access-dkv6k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:19:47.056805 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:47.056788    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7fd3273-aa8d-466e-a0e6-ec57f8723db6-maas-api-xks-ca" pod "a7fd3273-aa8d-466e-a0e6-ec57f8723db6" (UID: "a7fd3273-aa8d-466e-a0e6-ec57f8723db6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:19:47.156487 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:47.156418    4477 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7fd3273-aa8d-466e-a0e6-ec57f8723db6-maas-api-xks-ca\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:19:47.156487 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:47.156445    4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkv6k\" (UniqueName: \"kubernetes.io/projected/a7fd3273-aa8d-466e-a0e6-ec57f8723db6-kube-api-access-dkv6k\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:19:47.890957 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:47.890927    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d9b4df1-xrwbz"
Oct 30 22:19:47.891148 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:47.890965    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d9b4df1-xrwbz" event={"ID":"a7fd3273-aa8d-466e-a0e6-ec57f8723db6","Type":"ContainerDied","Data":"88e99f2043c2b3cbe5ad958afd11f999cf66deb94095bd166bb8392cc51ce894"}
Oct 30 22:19:47.891148 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:47.891012    4477 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="88e99f2043c2b3cbe5ad958afd11f999cf66deb94095bd166bb8392cc51ce894"
Oct 30 22:19:49.227936 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:49.227817    4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-d52c59/distinct-w3-d52c59-kserve-ffb5f5b96-l58xl"
Oct 30 22:19:49.241760 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:49.241733    4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-d52c59/distinct-w3-d52c59-kserve-ffb5f5b96-l58xl"
Oct 30 22:19:49.912886 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:49.912856    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d52c59/distinct-w3-d52c59-kserve-ffb5f5b96-l58xl"
Oct 30 22:19:56.269618 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:19:56.269581    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 22:19:58.924149 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:58.924115    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-421583/distinct-w2-421583-kserve-69cf5f4d6b-qbgnt"]
Oct 30 22:19:58.924576 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:58.924561    4477 cpu_manager.go:414] "containerMap: removing container" podUID="a7fd3273-aa8d-466e-a0e6-ec57f8723db6" containerName="revoke-keys"
Oct 30 22:19:58.924576 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:58.924577    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7fd3273-aa8d-466e-a0e6-ec57f8723db6" containerName="revoke-keys"
Oct 30 22:19:58.924677 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:58.924663    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7fd3273-aa8d-466e-a0e6-ec57f8723db6" containerName="revoke-keys"
Oct 30 22:19:58.929168 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:58.929144    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-421583/distinct-w2-421583-kserve-69cf5f4d6b-qbgnt"
Oct 30 22:19:58.948023 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:58.947999    4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-421583\"/\"openshift-service-ca.crt\""
Oct 30 22:19:58.954389 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:58.954368    4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-421583\"/\"kube-root-ca.crt\""
Oct 30 22:19:58.954639 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:58.954614    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-421583\"/\"distinct-w2-421583-kserve-self-signed-certs\""
Oct 30 22:19:58.954745 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:58.954645    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-421583\"/\"default-dockercfg-vhmms\""
Oct 30 22:19:58.961775 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:58.961751    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-421583/distinct-w2-421583-kserve-69cf5f4d6b-qbgnt"]
Oct 30 22:19:58.976089 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:58.976065    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/510ad10d-0554-47c8-9b55-deb359795071-model-cache\") pod \"distinct-w2-421583-kserve-69cf5f4d6b-qbgnt\" (UID: \"510ad10d-0554-47c8-9b55-deb359795071\") " pod="e2e-models-e2e-worker-w2-421583/distinct-w2-421583-kserve-69cf5f4d6b-qbgnt"
Oct 30 22:19:58.976315 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:58.976298    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/510ad10d-0554-47c8-9b55-deb359795071-tmp-dir\") pod \"distinct-w2-421583-kserve-69cf5f4d6b-qbgnt\" (UID: \"510ad10d-0554-47c8-9b55-deb359795071\") " pod="e2e-models-e2e-worker-w2-421583/distinct-w2-421583-kserve-69cf5f4d6b-qbgnt"
Oct 30 22:19:58.976475 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:58.976462    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/510ad10d-0554-47c8-9b55-deb359795071-dshm\") pod \"distinct-w2-421583-kserve-69cf5f4d6b-qbgnt\" (UID: \"510ad10d-0554-47c8-9b55-deb359795071\") " pod="e2e-models-e2e-worker-w2-421583/distinct-w2-421583-kserve-69cf5f4d6b-qbgnt"
Oct 30 22:19:58.976582 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:58.976567    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/510ad10d-0554-47c8-9b55-deb359795071-home\") pod \"distinct-w2-421583-kserve-69cf5f4d6b-qbgnt\" (UID: \"510ad10d-0554-47c8-9b55-deb359795071\") " pod="e2e-models-e2e-worker-w2-421583/distinct-w2-421583-kserve-69cf5f4d6b-qbgnt"
Oct 30 22:19:58.976710 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:58.976676    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/510ad10d-0554-47c8-9b55-deb359795071-tls-certs\") pod \"distinct-w2-421583-kserve-69cf5f4d6b-qbgnt\" (UID: \"510ad10d-0554-47c8-9b55-deb359795071\") " pod="e2e-models-e2e-worker-w2-421583/distinct-w2-421583-kserve-69cf5f4d6b-qbgnt"
Oct 30 22:19:58.976831 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:58.976817    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flh8t\" (UniqueName: \"kubernetes.io/projected/510ad10d-0554-47c8-9b55-deb359795071-kube-api-access-flh8t\") pod \"distinct-w2-421583-kserve-69cf5f4d6b-qbgnt\" (UID: \"510ad10d-0554-47c8-9b55-deb359795071\") " pod="e2e-models-e2e-worker-w2-421583/distinct-w2-421583-kserve-69cf5f4d6b-qbgnt"
Oct 30 22:19:59.077972 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:59.077930    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/510ad10d-0554-47c8-9b55-deb359795071-dshm\") pod \"distinct-w2-421583-kserve-69cf5f4d6b-qbgnt\" (UID: \"510ad10d-0554-47c8-9b55-deb359795071\") " pod="e2e-models-e2e-worker-w2-421583/distinct-w2-421583-kserve-69cf5f4d6b-qbgnt"
Oct 30 22:19:59.077972 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:59.077976    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/510ad10d-0554-47c8-9b55-deb359795071-home\") pod \"distinct-w2-421583-kserve-69cf5f4d6b-qbgnt\" (UID: \"510ad10d-0554-47c8-9b55-deb359795071\") " pod="e2e-models-e2e-worker-w2-421583/distinct-w2-421583-kserve-69cf5f4d6b-qbgnt"
Oct 30 22:19:59.078235 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:59.078019    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/510ad10d-0554-47c8-9b55-deb359795071-tls-certs\") pod \"distinct-w2-421583-kserve-69cf5f4d6b-qbgnt\" (UID: \"510ad10d-0554-47c8-9b55-deb359795071\") " pod="e2e-models-e2e-worker-w2-421583/distinct-w2-421583-kserve-69cf5f4d6b-qbgnt"
Oct 30 22:19:59.078235 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:59.078082    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flh8t\" (UniqueName: \"kubernetes.io/projected/510ad10d-0554-47c8-9b55-deb359795071-kube-api-access-flh8t\") pod \"distinct-w2-421583-kserve-69cf5f4d6b-qbgnt\" (UID: \"510ad10d-0554-47c8-9b55-deb359795071\") " pod="e2e-models-e2e-worker-w2-421583/distinct-w2-421583-kserve-69cf5f4d6b-qbgnt"
Oct 30 22:19:59.078235 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:59.078180    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/510ad10d-0554-47c8-9b55-deb359795071-model-cache\") pod \"distinct-w2-421583-kserve-69cf5f4d6b-qbgnt\" (UID: \"510ad10d-0554-47c8-9b55-deb359795071\") " pod="e2e-models-e2e-worker-w2-421583/distinct-w2-421583-kserve-69cf5f4d6b-qbgnt"
Oct 30 22:19:59.078235 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:59.078214    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/510ad10d-0554-47c8-9b55-deb359795071-tmp-dir\") pod \"distinct-w2-421583-kserve-69cf5f4d6b-qbgnt\" (UID: \"510ad10d-0554-47c8-9b55-deb359795071\") " pod="e2e-models-e2e-worker-w2-421583/distinct-w2-421583-kserve-69cf5f4d6b-qbgnt"
Oct 30 22:19:59.078447 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:59.078421    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/510ad10d-0554-47c8-9b55-deb359795071-home\") pod \"distinct-w2-421583-kserve-69cf5f4d6b-qbgnt\" (UID: \"510ad10d-0554-47c8-9b55-deb359795071\") " pod="e2e-models-e2e-worker-w2-421583/distinct-w2-421583-kserve-69cf5f4d6b-qbgnt"
Oct 30 22:19:59.078582 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:59.078553    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/510ad10d-0554-47c8-9b55-deb359795071-tmp-dir\") pod \"distinct-w2-421583-kserve-69cf5f4d6b-qbgnt\" (UID: \"510ad10d-0554-47c8-9b55-deb359795071\") " pod="e2e-models-e2e-worker-w2-421583/distinct-w2-421583-kserve-69cf5f4d6b-qbgnt"
Oct 30 22:19:59.078738 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:59.078711    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/510ad10d-0554-47c8-9b55-deb359795071-model-cache\") pod \"distinct-w2-421583-kserve-69cf5f4d6b-qbgnt\" (UID: \"510ad10d-0554-47c8-9b55-deb359795071\") " pod="e2e-models-e2e-worker-w2-421583/distinct-w2-421583-kserve-69cf5f4d6b-qbgnt"
Oct 30 22:19:59.080707 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:59.080687    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/510ad10d-0554-47c8-9b55-deb359795071-tls-certs\") pod \"distinct-w2-421583-kserve-69cf5f4d6b-qbgnt\" (UID: \"510ad10d-0554-47c8-9b55-deb359795071\") " pod="e2e-models-e2e-worker-w2-421583/distinct-w2-421583-kserve-69cf5f4d6b-qbgnt"
Oct 30 22:19:59.080859 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:59.080838    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/510ad10d-0554-47c8-9b55-deb359795071-dshm\") pod \"distinct-w2-421583-kserve-69cf5f4d6b-qbgnt\" (UID: \"510ad10d-0554-47c8-9b55-deb359795071\") " pod="e2e-models-e2e-worker-w2-421583/distinct-w2-421583-kserve-69cf5f4d6b-qbgnt"
Oct 30 22:19:59.096340 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:59.096318    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flh8t\" (UniqueName: \"kubernetes.io/projected/510ad10d-0554-47c8-9b55-deb359795071-kube-api-access-flh8t\") pod \"distinct-w2-421583-kserve-69cf5f4d6b-qbgnt\" (UID: \"510ad10d-0554-47c8-9b55-deb359795071\") " pod="e2e-models-e2e-worker-w2-421583/distinct-w2-421583-kserve-69cf5f4d6b-qbgnt"
Oct 30 22:19:59.241918 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:59.241835    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-421583/distinct-w2-421583-kserve-69cf5f4d6b-qbgnt"
Oct 30 22:19:59.544081 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:59.543983    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-421583/distinct-w2-421583-kserve-69cf5f4d6b-qbgnt"]
Oct 30 22:19:59.552599 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:19:59.552571    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod510ad10d_0554_47c8_9b55_deb359795071.slice/crio-3404620da8b739cff56a3be61466701871d2400bd57f2d82fc434a48e35134ca WatchSource:0}: Error finding container 3404620da8b739cff56a3be61466701871d2400bd57f2d82fc434a48e35134ca: Status 404 returned error can't find the container with id 3404620da8b739cff56a3be61466701871d2400bd57f2d82fc434a48e35134ca
Oct 30 22:19:59.960945 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:59.960907    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-421583/distinct-w2-421583-kserve-69cf5f4d6b-qbgnt" event={"ID":"510ad10d-0554-47c8-9b55-deb359795071","Type":"ContainerStarted","Data":"63dfc3553ebbd4e7044838049dc17fbae884a6d5b708b0a5763a81f774fbb1fd"}
Oct 30 22:19:59.960945 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:59.960951    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-421583/distinct-w2-421583-kserve-69cf5f4d6b-qbgnt" event={"ID":"510ad10d-0554-47c8-9b55-deb359795071","Type":"ContainerStarted","Data":"3404620da8b739cff56a3be61466701871d2400bd57f2d82fc434a48e35134ca"}
Oct 30 22:19:59.996464 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:19:59.996415    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-421583/distinct-w2-421583-kserve-69cf5f4d6b-qbgnt" podStartSLOduration=1.996398474 podStartE2EDuration="1.996398474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:19:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:19:59.996172003 +0000 UTC m=+783.922098226" watchObservedRunningTime="2026-10-30 22:19:59.996398474 +0000 UTC m=+783.922324684"
Oct 30 22:20:00.241945 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:00.241903    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-421583/distinct-w2-421583-kserve-69cf5f4d6b-qbgnt"
Oct 30 22:20:02.454673 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:02.454638    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6bz92"]
Oct 30 22:20:07.398388 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:07.398356    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-7m76p"]
Oct 30 22:20:07.403414 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:07.403395    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-7m76p"
Oct 30 22:20:07.427246 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:07.427228    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-d52c59-openshift-default-dockercfg-x7nqp\""
Oct 30 22:20:07.433727 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:07.433705    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-7m76p"]
Oct 30 22:20:07.503718 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:07.503692    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-kvtw4"]
Oct 30 22:20:07.507504 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:07.507483    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-kvtw4"
Oct 30 22:20:07.528204 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:07.528182    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-54114e-openshift-default-dockercfg-x4lxn\""
Oct 30 22:20:07.536891 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:07.536869    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-kvtw4"]
Oct 30 22:20:07.561638 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:07.561609    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0477d090-6e4c-4599-b4e9-b8e103c67fd1-workload-certs\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-7m76p\" (UID: \"0477d090-6e4c-4599-b4e9-b8e103c67fd1\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-7m76p"
Oct 30 22:20:07.561762 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:07.561655    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0477d090-6e4c-4599-b4e9-b8e103c67fd1-credential-socket\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-7m76p\" (UID: \"0477d090-6e4c-4599-b4e9-b8e103c67fd1\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-7m76p"
Oct 30 22:20:07.561806 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:07.561765    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0477d090-6e4c-4599-b4e9-b8e103c67fd1-istio-podinfo\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-7m76p\" (UID: \"0477d090-6e4c-4599-b4e9-b8e103c67fd1\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-7m76p"
Oct 30 22:20:07.561874 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:07.561859    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0477d090-6e4c-4599-b4e9-b8e103c67fd1-istio-envoy\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-7m76p\" (UID: \"0477d090-6e4c-4599-b4e9-b8e103c67fd1\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-7m76p"
Oct 30 22:20:07.562008 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:07.561985    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0477d090-6e4c-4599-b4e9-b8e103c67fd1-istio-data\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-7m76p\" (UID: \"0477d090-6e4c-4599-b4e9-b8e103c67fd1\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-7m76p"
Oct 30 22:20:07.562109 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:07.562089    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0477d090-6e4c-4599-b4e9-b8e103c67fd1-workload-socket\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-7m76p\" (UID: \"0477d090-6e4c-4599-b4e9-b8e103c67fd1\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-7m76p"
Oct 30 22:20:07.562221 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:07.562135    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqwkh\" (UniqueName: \"kubernetes.io/projected/0477d090-6e4c-4599-b4e9-b8e103c67fd1-kube-api-access-sqwkh\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-7m76p\" (UID: \"0477d090-6e4c-4599-b4e9-b8e103c67fd1\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-7m76p"
Oct 30 22:20:07.562221 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:07.562207    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0477d090-6e4c-4599-b4e9-b8e103c67fd1-istio-token\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-7m76p\" (UID: \"0477d090-6e4c-4599-b4e9-b8e103c67fd1\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-7m76p"
Oct 30 22:20:07.562310 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:07.562288    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0477d090-6e4c-4599-b4e9-b8e103c67fd1-istiod-ca-cert\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-7m76p\" (UID: \"0477d090-6e4c-4599-b4e9-b8e103c67fd1\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-7m76p"
Oct 30 22:20:07.654647 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:07.654578    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hq9vz"]
Oct 30 22:20:07.658653 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:07.658636    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hq9vz"
Oct 30 22:20:07.662955 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:07.662935    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a99ab39-789a-4cbf-8db5-5e85bfe485c2-workload-socket\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-kvtw4\" (UID: \"0a99ab39-789a-4cbf-8db5-5e85bfe485c2\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-kvtw4"
Oct 30 22:20:07.663085 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:07.662967    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0477d090-6e4c-4599-b4e9-b8e103c67fd1-istio-data\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-7m76p\" (UID: \"0477d090-6e4c-4599-b4e9-b8e103c67fd1\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-7m76p"
Oct 30 22:20:07.663085 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:07.663005    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0477d090-6e4c-4599-b4e9-b8e103c67fd1-workload-socket\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-7m76p\" (UID: \"0477d090-6e4c-4599-b4e9-b8e103c67fd1\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-7m76p"
Oct 30 22:20:07.663164 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:07.663104    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqwkh\" (UniqueName: \"kubernetes.io/projected/0477d090-6e4c-4599-b4e9-b8e103c67fd1-kube-api-access-sqwkh\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-7m76p\" (UID: \"0477d090-6e4c-4599-b4e9-b8e103c67fd1\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-7m76p"
Oct 30 22:20:07.663164 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:07.663132    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0477d090-6e4c-4599-b4e9-b8e103c67fd1-istio-token\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-7m76p\" (UID: \"0477d090-6e4c-4599-b4e9-b8e103c67fd1\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-7m76p"
Oct 30 22:20:07.663275 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:07.663251    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a99ab39-789a-4cbf-8db5-5e85bfe485c2-credential-socket\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-kvtw4\" (UID: \"0a99ab39-789a-4cbf-8db5-5e85bfe485c2\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-kvtw4"
Oct 30 22:20:07.663337 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:07.663305    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0477d090-6e4c-4599-b4e9-b8e103c67fd1-istiod-ca-cert\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-7m76p\" (UID: \"0477d090-6e4c-4599-b4e9-b8e103c67fd1\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-7m76p"
Oct 30 22:20:07.663392 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:07.663342    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0477d090-6e4c-4599-b4e9-b8e103c67fd1-istio-data\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-7m76p\" (UID: \"0477d090-6e4c-4599-b4e9-b8e103c67fd1\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-7m76p"
Oct 30 22:20:07.663455 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:07.663404    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0477d090-6e4c-4599-b4e9-b8e103c67fd1-workload-socket\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-7m76p\" (UID: \"0477d090-6e4c-4599-b4e9-b8e103c67fd1\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-7m76p"
Oct 30 22:20:07.663455 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:07.663411    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a99ab39-789a-4cbf-8db5-5e85bfe485c2-istio-podinfo\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-kvtw4\" (UID: \"0a99ab39-789a-4cbf-8db5-5e85bfe485c2\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-kvtw4"
Oct 30 22:20:07.663564 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:07.663471    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0477d090-6e4c-4599-b4e9-b8e103c67fd1-workload-certs\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-7m76p\" (UID: \"0477d090-6e4c-4599-b4e9-b8e103c67fd1\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-7m76p"
Oct 30 22:20:07.663564 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:07.663502    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a99ab39-789a-4cbf-8db5-5e85bfe485c2-istiod-ca-cert\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-kvtw4\" (UID: \"0a99ab39-789a-4cbf-8db5-5e85bfe485c2\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-kvtw4"
Oct 30 22:20:07.663564 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:07.663528    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0477d090-6e4c-4599-b4e9-b8e103c67fd1-credential-socket\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-7m76p\" (UID: \"0477d090-6e4c-4599-b4e9-b8e103c67fd1\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-7m76p"
Oct 30 22:20:07.663671 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:07.663568    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0477d090-6e4c-4599-b4e9-b8e103c67fd1-istio-podinfo\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-7m76p\" (UID: \"0477d090-6e4c-4599-b4e9-b8e103c67fd1\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-7m76p"
Oct 30 22:20:07.663671 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:07.663626    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgc78\" (UniqueName: \"kubernetes.io/projected/0a99ab39-789a-4cbf-8db5-5e85bfe485c2-kube-api-access-hgc78\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-kvtw4\" (UID: \"0a99ab39-789a-4cbf-8db5-5e85bfe485c2\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-kvtw4"
Oct 30 22:20:07.663671 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:07.663663    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0477d090-6e4c-4599-b4e9-b8e103c67fd1-workload-certs\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-7m76p\" (UID: \"0477d090-6e4c-4599-b4e9-b8e103c67fd1\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-7m76p"
Oct 30 22:20:07.663761 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:07.663724    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a99ab39-789a-4cbf-8db5-5e85bfe485c2-istio-data\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-kvtw4\" (UID: \"0a99ab39-789a-4cbf-8db5-5e85bfe485c2\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-kvtw4"
Oct 30 22:20:07.663814 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:07.663762    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a99ab39-789a-4cbf-8db5-5e85bfe485c2-workload-certs\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-kvtw4\" (UID: \"0a99ab39-789a-4cbf-8db5-5e85bfe485c2\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-kvtw4"
Oct 30 22:20:07.663871 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:07.663818    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0477d090-6e4c-4599-b4e9-b8e103c67fd1-credential-socket\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-7m76p\" (UID: \"0477d090-6e4c-4599-b4e9-b8e103c67fd1\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-7m76p"
Oct 30 22:20:07.663871 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:07.663861    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0477d090-6e4c-4599-b4e9-b8e103c67fd1-istio-envoy\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-7m76p\" (UID: \"0477d090-6e4c-4599-b4e9-b8e103c67fd1\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-7m76p"
Oct 30 22:20:07.663980 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:07.663902    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a99ab39-789a-4cbf-8db5-5e85bfe485c2-istio-token\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-kvtw4\" (UID: \"0a99ab39-789a-4cbf-8db5-5e85bfe485c2\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-kvtw4"
Oct 30 22:20:07.663980 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:07.663916    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0477d090-6e4c-4599-b4e9-b8e103c67fd1-istiod-ca-cert\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-7m76p\" (UID: \"0477d090-6e4c-4599-b4e9-b8e103c67fd1\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-7m76p"
Oct 30 22:20:07.663980 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:07.663933    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a99ab39-789a-4cbf-8db5-5e85bfe485c2-istio-envoy\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-kvtw4\" (UID: \"0a99ab39-789a-4cbf-8db5-5e85bfe485c2\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-kvtw4"
Oct 30 22:20:07.666152 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:07.666131    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0477d090-6e4c-4599-b4e9-b8e103c67fd1-istio-podinfo\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-7m76p\" (UID: \"0477d090-6e4c-4599-b4e9-b8e103c67fd1\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-7m76p"
Oct 30 22:20:07.666469 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:07.666450    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0477d090-6e4c-4599-b4e9-b8e103c67fd1-istio-envoy\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-7m76p\" (UID: \"0477d090-6e4c-4599-b4e9-b8e103c67fd1\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-7m76p"
Oct 30 22:20:07.672201 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:07.672185    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-0ad7c5-openshift-default-dockercfg-b9j97\""
Oct 30 22:20:07.681417 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:07.681391    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hq9vz"]
Oct 30 22:20:07.683503 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:07.683448    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqwkh\" (UniqueName: \"kubernetes.io/projected/0477d090-6e4c-4599-b4e9-b8e103c67fd1-kube-api-access-sqwkh\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-7m76p\" (UID: \"0477d090-6e4c-4599-b4e9-b8e103c67fd1\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-7m76p"
Oct 30 22:20:07.683619 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:07.683522    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0477d090-6e4c-4599-b4e9-b8e103c67fd1-istio-token\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-7m76p\" (UID: \"0477d090-6e4c-4599-b4e9-b8e103c67fd1\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-7m76p"
Oct 30 22:20:07.718431 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:07.718409    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-7m76p"
Oct 30 22:20:07.764805 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:07.764771    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1-istio-podinfo\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hq9vz\" (UID: \"76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hq9vz"
Oct 30 22:20:07.764961 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:07.764855    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a99ab39-789a-4cbf-8db5-5e85bfe485c2-credential-socket\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-kvtw4\" (UID: \"0a99ab39-789a-4cbf-8db5-5e85bfe485c2\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-kvtw4"
Oct 30 22:20:07.764961 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:07.764910    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1-workload-socket\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hq9vz\" (UID: \"76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hq9vz"
Oct 30 22:20:07.764961 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:07.764940    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1-istiod-ca-cert\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hq9vz\" (UID: \"76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hq9vz"
Oct 30 22:20:07.765088 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:07.765036    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a99ab39-789a-4cbf-8db5-5e85bfe485c2-istio-podinfo\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-kvtw4\" (UID: \"0a99ab39-789a-4cbf-8db5-5e85bfe485c2\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-kvtw4"
Oct 30 22:20:07.765088 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:07.765078    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1-workload-certs\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hq9vz\" (UID: \"76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hq9vz"
Oct 30 22:20:07.765152 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:07.765129    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a99ab39-789a-4cbf-8db5-5e85bfe485c2-istiod-ca-cert\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-kvtw4\" (UID: \"0a99ab39-789a-4cbf-8db5-5e85bfe485c2\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-kvtw4"
Oct 30 22:20:07.765191 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:07.765179    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgc78\" (UniqueName: \"kubernetes.io/projected/0a99ab39-789a-4cbf-8db5-5e85bfe485c2-kube-api-access-hgc78\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-kvtw4\" (UID: \"0a99ab39-789a-4cbf-8db5-5e85bfe485c2\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-kvtw4"
Oct 30 22:20:07.765302 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:07.765245    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1-credential-socket\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hq9vz\" (UID: \"76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hq9vz"
Oct 30 22:20:07.765302 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:07.765264    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a99ab39-789a-4cbf-8db5-5e85bfe485c2-credential-socket\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-kvtw4\" (UID: \"0a99ab39-789a-4cbf-8db5-5e85bfe485c2\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-kvtw4"
Oct 30 22:20:07.765502 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:07.765324    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9whg\" (UniqueName: \"kubernetes.io/projected/76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1-kube-api-access-d9whg\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hq9vz\" (UID: \"76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hq9vz"
Oct 30 22:20:07.765502 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:07.765425    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1-istio-data\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hq9vz\" (UID: \"76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hq9vz"
Oct 30 22:20:07.765502 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:07.765478    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1-istio-envoy\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hq9vz\" (UID: \"76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hq9vz"
Oct 30 22:20:07.765620 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:07.765515    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a99ab39-789a-4cbf-8db5-5e85bfe485c2-istio-data\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-kvtw4\" (UID: \"0a99ab39-789a-4cbf-8db5-5e85bfe485c2\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-kvtw4"
Oct 30 22:20:07.765620 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:07.765583    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a99ab39-789a-4cbf-8db5-5e85bfe485c2-workload-certs\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-kvtw4\" (UID: \"0a99ab39-789a-4cbf-8db5-5e85bfe485c2\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-kvtw4"
Oct 30 22:20:07.765699 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:07.765626    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1-istio-token\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hq9vz\" (UID: \"76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hq9vz"
Oct 30 22:20:07.765756 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:07.765695    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a99ab39-789a-4cbf-8db5-5e85bfe485c2-istio-token\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-kvtw4\" (UID: \"0a99ab39-789a-4cbf-8db5-5e85bfe485c2\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-kvtw4"
Oct 30 22:20:07.765756 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:07.765737    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a99ab39-789a-4cbf-8db5-5e85bfe485c2-istio-envoy\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-kvtw4\" (UID: \"0a99ab39-789a-4cbf-8db5-5e85bfe485c2\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-kvtw4"
Oct 30 22:20:07.765857 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:07.765781    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a99ab39-789a-4cbf-8db5-5e85bfe485c2-workload-socket\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-kvtw4\" (UID: \"0a99ab39-789a-4cbf-8db5-5e85bfe485c2\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-kvtw4"
Oct 30 22:20:07.765857 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:07.765807    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a99ab39-789a-4cbf-8db5-5e85bfe485c2-istio-data\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-kvtw4\" (UID: \"0a99ab39-789a-4cbf-8db5-5e85bfe485c2\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-kvtw4"
Oct 30 22:20:07.765962 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:07.765941    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a99ab39-789a-4cbf-8db5-5e85bfe485c2-workload-certs\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-kvtw4\" (UID: \"0a99ab39-789a-4cbf-8db5-5e85bfe485c2\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-kvtw4"
Oct 30 22:20:07.766024 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:07.765998    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a99ab39-789a-4cbf-8db5-5e85bfe485c2-istiod-ca-cert\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-kvtw4\" (UID: \"0a99ab39-789a-4cbf-8db5-5e85bfe485c2\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-kvtw4"
Oct 30 22:20:07.766242 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:07.766217    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a99ab39-789a-4cbf-8db5-5e85bfe485c2-workload-socket\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-kvtw4\" (UID: \"0a99ab39-789a-4cbf-8db5-5e85bfe485c2\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-kvtw4"
Oct 30 22:20:07.767904 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:07.767837    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a99ab39-789a-4cbf-8db5-5e85bfe485c2-istio-podinfo\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-kvtw4\" (UID: \"0a99ab39-789a-4cbf-8db5-5e85bfe485c2\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-kvtw4"
Oct 30 22:20:07.768156 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:07.768120    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a99ab39-789a-4cbf-8db5-5e85bfe485c2-istio-envoy\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-kvtw4\" (UID: \"0a99ab39-789a-4cbf-8db5-5e85bfe485c2\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-kvtw4"
Oct 30 22:20:07.792929 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:07.791141    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgc78\" (UniqueName: \"kubernetes.io/projected/0a99ab39-789a-4cbf-8db5-5e85bfe485c2-kube-api-access-hgc78\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-kvtw4\" (UID: \"0a99ab39-789a-4cbf-8db5-5e85bfe485c2\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-kvtw4"
Oct 30 22:20:07.792929 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:07.792699    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a99ab39-789a-4cbf-8db5-5e85bfe485c2-istio-token\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-kvtw4\" (UID: \"0a99ab39-789a-4cbf-8db5-5e85bfe485c2\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-kvtw4"
Oct 30 22:20:07.797871 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:07.797528    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-lg99w"]
Oct 30 22:20:07.803306 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:07.802958    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-lg99w"
Oct 30 22:20:07.817067 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:07.816687    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-012ba0-openshift-default-dockercfg-gk2x2\""
Oct 30 22:20:07.820447 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:07.820430    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-kvtw4"
Oct 30 22:20:07.823781 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:07.823670    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-lg99w"]
Oct 30 22:20:07.866582 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:07.866547    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1-istio-envoy\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hq9vz\" (UID: \"76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hq9vz"
Oct 30 22:20:07.867083 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:07.867057    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1-istio-token\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hq9vz\" (UID: \"76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hq9vz"
Oct 30 22:20:07.867234 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:07.867217    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1-istio-podinfo\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hq9vz\" (UID: \"76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hq9vz"
Oct 30 22:20:07.867631 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:07.867612    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1-workload-socket\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hq9vz\" (UID: \"76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hq9vz"
Oct 30 22:20:07.867710 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:07.867636    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1-istiod-ca-cert\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hq9vz\" (UID: \"76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hq9vz"
Oct 30 22:20:07.867890 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:07.867864    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1-workload-socket\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hq9vz\" (UID: \"76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hq9vz"
Oct 30 22:20:07.867973 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:07.867958    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1-workload-certs\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hq9vz\" (UID: \"76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hq9vz"
Oct 30 22:20:07.868031 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:07.868007    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1-credential-socket\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hq9vz\" (UID: \"76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hq9vz"
Oct 30 22:20:07.868031 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:07.868022    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9whg\" (UniqueName: \"kubernetes.io/projected/76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1-kube-api-access-d9whg\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hq9vz\" (UID: \"76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hq9vz"
Oct 30 22:20:07.868265 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:07.868248    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1-credential-socket\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hq9vz\" (UID: \"76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hq9vz"
Oct 30 22:20:07.868380 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:07.868355    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1-workload-certs\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hq9vz\" (UID: \"76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hq9vz"
Oct 30 22:20:07.868458 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:07.868426    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1-istio-data\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hq9vz\" (UID: \"76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hq9vz"
Oct 30 22:20:07.868625 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:07.868607    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1-istio-data\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hq9vz\" (UID: \"76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hq9vz"
Oct 30 22:20:07.868982 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:07.868956    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1-istio-envoy\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hq9vz\" (UID: \"76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hq9vz"
Oct 30 22:20:07.869229 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:07.869205    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1-istiod-ca-cert\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hq9vz\" (UID: \"76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hq9vz"
Oct 30 22:20:07.869848 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:07.869828    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1-istio-podinfo\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hq9vz\" (UID: \"76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hq9vz"
Oct 30 22:20:07.887395 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:07.886733    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9whg\" (UniqueName: \"kubernetes.io/projected/76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1-kube-api-access-d9whg\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hq9vz\" (UID: \"76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hq9vz"
Oct 30 22:20:07.887801 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:07.887780    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1-istio-token\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hq9vz\" (UID: \"76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hq9vz"
Oct 30 22:20:07.969191 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:07.969128    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e7915fd-e76a-407d-825e-ae095f5d3c56-istio-data\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-lg99w\" (UID: \"8e7915fd-e76a-407d-825e-ae095f5d3c56\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-lg99w"
Oct 30 22:20:07.969191 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:07.969179    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e7915fd-e76a-407d-825e-ae095f5d3c56-istiod-ca-cert\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-lg99w\" (UID: \"8e7915fd-e76a-407d-825e-ae095f5d3c56\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-lg99w"
Oct 30 22:20:07.969342 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:07.969203    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e7915fd-e76a-407d-825e-ae095f5d3c56-istio-podinfo\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-lg99w\" (UID: \"8e7915fd-e76a-407d-825e-ae095f5d3c56\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-lg99w"
Oct 30 22:20:07.969342 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:07.969296    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e7915fd-e76a-407d-825e-ae095f5d3c56-istio-envoy\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-lg99w\" (UID: \"8e7915fd-e76a-407d-825e-ae095f5d3c56\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-lg99w"
Oct 30 22:20:07.969342 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:07.969326    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e7915fd-e76a-407d-825e-ae095f5d3c56-workload-socket\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-lg99w\" (UID: \"8e7915fd-e76a-407d-825e-ae095f5d3c56\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-lg99w"
Oct 30 22:20:07.969472 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:07.969417    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e7915fd-e76a-407d-825e-ae095f5d3c56-workload-certs\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-lg99w\" (UID: \"8e7915fd-e76a-407d-825e-ae095f5d3c56\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-lg99w"
Oct 30 22:20:07.969472 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:07.969450    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vkj7\" (UniqueName: \"kubernetes.io/projected/8e7915fd-e76a-407d-825e-ae095f5d3c56-kube-api-access-6vkj7\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-lg99w\" (UID: \"8e7915fd-e76a-407d-825e-ae095f5d3c56\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-lg99w"
Oct 30 22:20:07.969543 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:07.969526    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e7915fd-e76a-407d-825e-ae095f5d3c56-istio-token\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-lg99w\" (UID: \"8e7915fd-e76a-407d-825e-ae095f5d3c56\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-lg99w"
Oct 30 22:20:07.969579 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:07.969552    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e7915fd-e76a-407d-825e-ae095f5d3c56-credential-socket\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-lg99w\" (UID: \"8e7915fd-e76a-407d-825e-ae095f5d3c56\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-lg99w"
Oct 30 22:20:07.985304 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:07.985283    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hq9vz"
Oct 30 22:20:08.030602 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:08.030571    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-7m76p"]
Oct 30 22:20:08.035300 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:20:08.035275    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0477d090_6e4c_4599_b4e9_b8e103c67fd1.slice/crio-41f4023bfce3e7503666f38e3ecd1f18b70b7e5ad7122b4f6359c656f53b537a WatchSource:0}: Error finding container 41f4023bfce3e7503666f38e3ecd1f18b70b7e5ad7122b4f6359c656f53b537a: Status 404 returned error can't find the container with id 41f4023bfce3e7503666f38e3ecd1f18b70b7e5ad7122b4f6359c656f53b537a
Oct 30 22:20:08.038980 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:08.038570    4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:20:08.038980 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:08.038660    4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:20:08.038980 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:08.038705    4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:20:08.070262 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:08.070238    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e7915fd-e76a-407d-825e-ae095f5d3c56-istiod-ca-cert\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-lg99w\" (UID: \"8e7915fd-e76a-407d-825e-ae095f5d3c56\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-lg99w"
Oct 30 22:20:08.070371 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:08.070283    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e7915fd-e76a-407d-825e-ae095f5d3c56-istio-podinfo\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-lg99w\" (UID: \"8e7915fd-e76a-407d-825e-ae095f5d3c56\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-lg99w"
Oct 30 22:20:08.070371 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:08.070324    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e7915fd-e76a-407d-825e-ae095f5d3c56-istio-envoy\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-lg99w\" (UID: \"8e7915fd-e76a-407d-825e-ae095f5d3c56\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-lg99w"
Oct 30 22:20:08.070371 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:08.070352    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e7915fd-e76a-407d-825e-ae095f5d3c56-workload-socket\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-lg99w\" (UID: \"8e7915fd-e76a-407d-825e-ae095f5d3c56\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-lg99w"
Oct 30 22:20:08.070535 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:08.070431    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e7915fd-e76a-407d-825e-ae095f5d3c56-workload-certs\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-lg99w\" (UID: \"8e7915fd-e76a-407d-825e-ae095f5d3c56\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-lg99w"
Oct 30 22:20:08.070535 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:08.070463    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vkj7\" (UniqueName: \"kubernetes.io/projected/8e7915fd-e76a-407d-825e-ae095f5d3c56-kube-api-access-6vkj7\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-lg99w\" (UID: \"8e7915fd-e76a-407d-825e-ae095f5d3c56\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-lg99w"
Oct 30 22:20:08.070535 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:08.070529    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e7915fd-e76a-407d-825e-ae095f5d3c56-istio-token\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-lg99w\" (UID: \"8e7915fd-e76a-407d-825e-ae095f5d3c56\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-lg99w"
Oct 30 22:20:08.070687 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:08.070569    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e7915fd-e76a-407d-825e-ae095f5d3c56-credential-socket\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-lg99w\" (UID: \"8e7915fd-e76a-407d-825e-ae095f5d3c56\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-lg99w"
Oct 30 22:20:08.070687 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:08.070645    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e7915fd-e76a-407d-825e-ae095f5d3c56-istio-data\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-lg99w\" (UID: \"8e7915fd-e76a-407d-825e-ae095f5d3c56\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-lg99w"
Oct 30 22:20:08.070792 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:08.070739    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e7915fd-e76a-407d-825e-ae095f5d3c56-workload-socket\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-lg99w\" (UID: \"8e7915fd-e76a-407d-825e-ae095f5d3c56\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-lg99w"
Oct 30 22:20:08.070962 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:08.070943    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e7915fd-e76a-407d-825e-ae095f5d3c56-istio-data\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-lg99w\" (UID: \"8e7915fd-e76a-407d-825e-ae095f5d3c56\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-lg99w"
Oct 30 22:20:08.071027 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:08.070981    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e7915fd-e76a-407d-825e-ae095f5d3c56-istiod-ca-cert\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-lg99w\" (UID: \"8e7915fd-e76a-407d-825e-ae095f5d3c56\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-lg99w"
Oct 30 22:20:08.071320 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:08.071298    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e7915fd-e76a-407d-825e-ae095f5d3c56-workload-certs\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-lg99w\" (UID: \"8e7915fd-e76a-407d-825e-ae095f5d3c56\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-lg99w"
Oct 30 22:20:08.071387 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:08.071308    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e7915fd-e76a-407d-825e-ae095f5d3c56-credential-socket\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-lg99w\" (UID: \"8e7915fd-e76a-407d-825e-ae095f5d3c56\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-lg99w"
Oct 30 22:20:08.072940 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:08.072918    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e7915fd-e76a-407d-825e-ae095f5d3c56-istio-envoy\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-lg99w\" (UID: \"8e7915fd-e76a-407d-825e-ae095f5d3c56\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-lg99w"
Oct 30 22:20:08.073020 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:08.072967    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e7915fd-e76a-407d-825e-ae095f5d3c56-istio-podinfo\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-lg99w\" (UID: \"8e7915fd-e76a-407d-825e-ae095f5d3c56\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-lg99w"
Oct 30 22:20:08.089267 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:08.089242    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e7915fd-e76a-407d-825e-ae095f5d3c56-istio-token\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-lg99w\" (UID: \"8e7915fd-e76a-407d-825e-ae095f5d3c56\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-lg99w"
Oct 30 22:20:08.089381 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:08.089366    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vkj7\" (UniqueName: \"kubernetes.io/projected/8e7915fd-e76a-407d-825e-ae095f5d3c56-kube-api-access-6vkj7\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-lg99w\" (UID: \"8e7915fd-e76a-407d-825e-ae095f5d3c56\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-lg99w"
Oct 30 22:20:08.118875 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:08.118854    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-lg99w"
Oct 30 22:20:08.129325 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:08.129304    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-kvtw4"]
Oct 30 22:20:08.132121 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:20:08.132093    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a99ab39_789a_4cbf_8db5_5e85bfe485c2.slice/crio-82c270fdc8137654099e656f24b6b55b29d2703900753f0b26663480ef419789 WatchSource:0}: Error finding container 82c270fdc8137654099e656f24b6b55b29d2703900753f0b26663480ef419789: Status 404 returned error can't find the container with id 82c270fdc8137654099e656f24b6b55b29d2703900753f0b26663480ef419789
Oct 30 22:20:08.134658 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:08.134625    4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:20:08.134757 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:08.134688    4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:20:08.134757 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:08.134716    4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:20:08.289621 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:08.289573    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hq9vz"]
Oct 30 22:20:08.294467 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:20:08.294439    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76fb88a8_b2ba_42c2_99c2_89fb9fbd4ca1.slice/crio-728439bf6521aeaca021a03e1dc4f70536707ee943306117437a01fca6c34d9c WatchSource:0}: Error finding container 728439bf6521aeaca021a03e1dc4f70536707ee943306117437a01fca6c34d9c: Status 404 returned error can't find the container with id 728439bf6521aeaca021a03e1dc4f70536707ee943306117437a01fca6c34d9c
Oct 30 22:20:08.297512 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:08.297023    4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:20:08.297512 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:08.297141    4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:20:08.297512 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:08.297181    4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:20:08.451348 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:08.450826    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-lg99w"]
Oct 30 22:20:08.455160 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:20:08.455126    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e7915fd_e76a_407d_825e_ae095f5d3c56.slice/crio-58654c4afab4ae6b6d6811b7e4a553a3cd1ece0844548bbc8169da4d7090a23d WatchSource:0}: Error finding container 58654c4afab4ae6b6d6811b7e4a553a3cd1ece0844548bbc8169da4d7090a23d: Status 404 returned error can't find the container with id 58654c4afab4ae6b6d6811b7e4a553a3cd1ece0844548bbc8169da4d7090a23d
Oct 30 22:20:08.457749 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:08.457709    4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:20:08.457839 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:08.457792    4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:20:08.457878 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:08.457835    4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:20:09.068747 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:09.067255    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hq9vz" event={"ID":"76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1","Type":"ContainerStarted","Data":"db69e1b6f0249022202674e28dc9b17682d88f0154d4960524be464b4e7046e4"}
Oct 30 22:20:09.068747 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:09.067367    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hq9vz" event={"ID":"76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1","Type":"ContainerStarted","Data":"728439bf6521aeaca021a03e1dc4f70536707ee943306117437a01fca6c34d9c"}
Oct 30 22:20:09.108223 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:09.108183    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-lg99w" event={"ID":"8e7915fd-e76a-407d-825e-ae095f5d3c56","Type":"ContainerStarted","Data":"de5712116b50a298f8a609b6ca5f0b7b67659e2cfd6c70cb4060fe984f6318bc"}
Oct 30 22:20:09.108465 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:09.108447    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-lg99w" event={"ID":"8e7915fd-e76a-407d-825e-ae095f5d3c56","Type":"ContainerStarted","Data":"58654c4afab4ae6b6d6811b7e4a553a3cd1ece0844548bbc8169da4d7090a23d"}
Oct 30 22:20:09.123081 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:09.123022    4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-lg99w"
Oct 30 22:20:09.123353 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:09.123328    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-kvtw4" event={"ID":"0a99ab39-789a-4cbf-8db5-5e85bfe485c2","Type":"ContainerStarted","Data":"df69e97adfaae616164d5729ef54b4bb176685252fac4c457de03fd8bb03bec9"}
Oct 30 22:20:09.123507 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:09.123488    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-kvtw4" event={"ID":"0a99ab39-789a-4cbf-8db5-5e85bfe485c2","Type":"ContainerStarted","Data":"82c270fdc8137654099e656f24b6b55b29d2703900753f0b26663480ef419789"}
Oct 30 22:20:09.136645 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:09.136609    4477 patch_prober.go:29] interesting pod/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-lg99w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Oct 30 22:20:09.136903 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:09.136876    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-lg99w" podUID="8e7915fd-e76a-407d-825e-ae095f5d3c56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Oct 30 22:20:09.139480 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:09.139451    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-7m76p" event={"ID":"0477d090-6e4c-4599-b4e9-b8e103c67fd1","Type":"ContainerStarted","Data":"d12a0d10159da837f2cd37c125415bf1d92195a8858f12b0378a995bd11e3fbc"}
Oct 30 22:20:09.139650 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:09.139636    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-7m76p" event={"ID":"0477d090-6e4c-4599-b4e9-b8e103c67fd1","Type":"ContainerStarted","Data":"41f4023bfce3e7503666f38e3ecd1f18b70b7e5ad7122b4f6359c656f53b537a"}
Oct 30 22:20:09.241420 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:09.241347    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hq9vz" podStartSLOduration=2.241327436 podStartE2EDuration="2.241327436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:20:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:20:09.154397033 +0000 UTC m=+793.080323264" watchObservedRunningTime="2026-10-30 22:20:09.241327436 +0000 UTC m=+793.167253643"
Oct 30 22:20:09.246063 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:09.241800    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-kvtw4" podStartSLOduration=2.241790294 podStartE2EDuration="2.241790294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:20:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:20:09.221376016 +0000 UTC m=+793.147302249" watchObservedRunningTime="2026-10-30 22:20:09.241790294 +0000 UTC m=+793.167716501"
Oct 30 22:20:09.246063 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:09.244137    4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-421583/distinct-w2-421583-kserve-69cf5f4d6b-qbgnt"
Oct 30 22:20:09.304068 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:09.303790    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-lg99w" podStartSLOduration=2.303771122 podStartE2EDuration="2.303771122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:20:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:20:09.300556593 +0000 UTC m=+793.226482803" watchObservedRunningTime="2026-10-30 22:20:09.303771122 +0000 UTC m=+793.229697330"
Oct 30 22:20:09.367120 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:09.364395    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-7m76p" podStartSLOduration=2.364376145 podStartE2EDuration="2.364376145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:20:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:20:09.359735748 +0000 UTC m=+793.285661977" watchObservedRunningTime="2026-10-30 22:20:09.364376145 +0000 UTC m=+793.290302353"
Oct 30 22:20:09.734454 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:09.734413    4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-7m76p"
Oct 30 22:20:09.738873 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:09.738840    4477 patch_prober.go:29] interesting pod/e2e-worker-w3-d52c59-openshift-default-5466968fb7-7m76p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Oct 30 22:20:09.739107 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:09.739078    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-7m76p" podUID="0477d090-6e4c-4599-b4e9-b8e103c67fd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Oct 30 22:20:09.829908 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:09.829869    4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-kvtw4"
Oct 30 22:20:09.830272 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:09.830250    4477 patch_prober.go:29] interesting pod/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-kvtw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Oct 30 22:20:09.830439 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:09.830411    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-kvtw4" podUID="0a99ab39-789a-4cbf-8db5-5e85bfe485c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Oct 30 22:20:09.958066 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:09.957430    4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-421583/distinct-w2-421583-kserve-69cf5f4d6b-qbgnt"
Oct 30 22:20:09.990123 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:09.989865    4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hq9vz"
Oct 30 22:20:09.990123 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:09.990008    4477 patch_prober.go:29] interesting pod/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hq9vz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Oct 30 22:20:09.990123 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:09.990080    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hq9vz" podUID="76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Oct 30 22:20:10.120065 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:10.119522    4477 patch_prober.go:29] interesting pod/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-lg99w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Oct 30 22:20:10.120065 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:10.119601    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-lg99w" podUID="8e7915fd-e76a-407d-825e-ae095f5d3c56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Oct 30 22:20:10.239066 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:10.238378    4477 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-d52c59/distinct-w3-d52c59-kserve-ffb5f5b96-l58xl" podUID="c7b1483b-2631-4aa2-a888-3096761d5494" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": context deadline exceeded"
Oct 30 22:20:10.427067 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:10.425146    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0ad7c5/premium-w4-0ad7c5-kserve-7bbf78cd4-pm6wc" podUID="5c10371e-56e7-42ec-9ef8-0802e15cb2ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:20:10.456740 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:10.456141    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/premium-w5-54114e-kserve-57f6747bcc-7wjrw" podUID="61990d22-ab8f-45bc-a4d6-4c8d528742f1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:20:10.457769 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:10.457461    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/premium-w6-012ba0-kserve-658654dbdb-d2k27" podUID="085847e5-4f50-419e-984f-b612b104ed90" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded"
Oct 30 22:20:10.730123 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:10.729178    4477 patch_prober.go:29] interesting pod/e2e-worker-w3-d52c59-openshift-default-5466968fb7-7m76p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Oct 30 22:20:10.730123 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:10.729242    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-7m76p" podUID="0477d090-6e4c-4599-b4e9-b8e103c67fd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Oct 30 22:20:10.770066 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:10.769345    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-421583/distinct-w2-421583-kserve-69cf5f4d6b-qbgnt"
Oct 30 22:20:10.837065 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:10.828220    4477 patch_prober.go:29] interesting pod/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-kvtw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Oct 30 22:20:10.837065 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:10.828280    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-kvtw4" podUID="0a99ab39-789a-4cbf-8db5-5e85bfe485c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Oct 30 22:20:11.072801 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:11.051281    4477 patch_prober.go:29] interesting pod/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hq9vz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Oct 30 22:20:11.072801 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:11.051376    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hq9vz" podUID="76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Oct 30 22:20:11.123623 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:11.123199    4477 patch_prober.go:29] interesting pod/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-lg99w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Oct 30 22:20:11.123623 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:11.123270    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-lg99w" podUID="8e7915fd-e76a-407d-825e-ae095f5d3c56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Oct 30 22:20:11.464065 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:11.463717    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/premium-w5-54114e-kserve-57f6747bcc-7wjrw" podUID="61990d22-ab8f-45bc-a4d6-4c8d528742f1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:20:11.468086 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:11.464537    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-54114e/premium-w5-54114e-kserve-57f6747bcc-7wjrw"
Oct 30 22:20:11.727120 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:11.721899    4477 patch_prober.go:29] interesting pod/e2e-worker-w3-d52c59-openshift-default-5466968fb7-7m76p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Oct 30 22:20:11.727120 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:11.721958    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-7m76p" podUID="0477d090-6e4c-4599-b4e9-b8e103c67fd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Oct 30 22:20:11.835667 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:11.835626    4477 patch_prober.go:29] interesting pod/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-kvtw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Oct 30 22:20:11.838025 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:11.837991    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-kvtw4" podUID="0a99ab39-789a-4cbf-8db5-5e85bfe485c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Oct 30 22:20:11.855702 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:11.853113    4477 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-54114e/premium-w5-54114e-kserve-57f6747bcc-7wjrw" podUID="61990d22-ab8f-45bc-a4d6-4c8d528742f1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:20:11.989113 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:11.988207    4477 patch_prober.go:29] interesting pod/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hq9vz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Oct 30 22:20:11.989113 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:11.988265    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hq9vz" podUID="76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Oct 30 22:20:11.997715 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:11.997688    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-54114e/premium-w5-54114e-kserve-57f6747bcc-7wjrw"
Oct 30 22:20:12.150490 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:12.150459    4477 patch_prober.go:29] interesting pod/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-lg99w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Oct 30 22:20:12.150750 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:12.150726    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-lg99w" podUID="8e7915fd-e76a-407d-825e-ae095f5d3c56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Oct 30 22:20:12.238066 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:12.235768    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/distinct-w2-421583-kserve-69cf5f4d6b-qbgnt" podUID="510ad10d-0554-47c8-9b55-deb359795071" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:20:12.238066 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:12.236085    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/simulator-w1-2b1e88-kserve-6c579b99f8-q5wpq" podUID="722f10d8-b7a2-4bf7-a274-9669529ef427" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded"
Oct 30 22:20:12.415838 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:12.415797    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0ad7c5/premium-w4-0ad7c5-kserve-7bbf78cd4-pm6wc" podUID="5c10371e-56e7-42ec-9ef8-0802e15cb2ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:20:12.464670 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:12.464624    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/premium-w6-012ba0-kserve-658654dbdb-d2k27" podUID="085847e5-4f50-419e-984f-b612b104ed90" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded"
Oct 30 22:20:12.728121 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:12.727682    4477 patch_prober.go:29] interesting pod/e2e-worker-w3-d52c59-openshift-default-5466968fb7-7m76p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Oct 30 22:20:12.728121 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:12.727757    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-7m76p" podUID="0477d090-6e4c-4599-b4e9-b8e103c67fd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Oct 30 22:20:12.830067 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:12.821545    4477 patch_prober.go:29] interesting pod/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-kvtw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Oct 30 22:20:12.830067 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:12.821611    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-kvtw4" podUID="0a99ab39-789a-4cbf-8db5-5e85bfe485c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Oct 30 22:20:12.991119 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:12.988361    4477 patch_prober.go:29] interesting pod/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hq9vz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Oct 30 22:20:12.991119 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:12.988430    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hq9vz" podUID="76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Oct 30 22:20:13.123349 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:13.123310    4477 patch_prober.go:29] interesting pod/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-lg99w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Oct 30 22:20:13.123729 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:13.123699    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-lg99w" podUID="8e7915fd-e76a-407d-825e-ae095f5d3c56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Oct 30 22:20:13.243705 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:13.243611    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/distinct-w2-421583-kserve-69cf5f4d6b-qbgnt" podUID="510ad10d-0554-47c8-9b55-deb359795071" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:20:13.244649 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:13.244627    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-421583/distinct-w2-421583-kserve-69cf5f4d6b-qbgnt"
Oct 30 22:20:13.473052 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:13.472995    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0ad7c5/premium-w4-0ad7c5-kserve-7bbf78cd4-pm6wc" podUID="5c10371e-56e7-42ec-9ef8-0802e15cb2ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:20:13.534638 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:13.534555    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0ad7c5/premium-w4-0ad7c5-kserve-7bbf78cd4-pm6wc"
Oct 30 22:20:13.769496 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:13.769461    4477 patch_prober.go:29] interesting pod/e2e-worker-w3-d52c59-openshift-default-5466968fb7-7m76p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Oct 30 22:20:13.769991 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:13.769961    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-7m76p" podUID="0477d090-6e4c-4599-b4e9-b8e103c67fd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Oct 30 22:20:13.825933 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:13.825837    4477 patch_prober.go:29] interesting pod/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-kvtw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Oct 30 22:20:13.830171 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:13.826176    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-kvtw4" podUID="0a99ab39-789a-4cbf-8db5-5e85bfe485c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Oct 30 22:20:13.918513 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:13.918454    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/distinct-w3-d52c59-kserve-ffb5f5b96-l58xl" podUID="c7b1483b-2631-4aa2-a888-3096761d5494" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:20:13.987633 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:13.987587    4477 patch_prober.go:29] interesting pod/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hq9vz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Oct 30 22:20:13.987924 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:13.987885    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hq9vz" podUID="76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Oct 30 22:20:14.011719 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:14.011682    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/premium-w5-54114e-kserve-57f6747bcc-7wjrw" podUID="61990d22-ab8f-45bc-a4d6-4c8d528742f1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:20:14.129180 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:14.129145    4477 patch_prober.go:29] interesting pod/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-lg99w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Oct 30 22:20:14.129454 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:14.129425    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-lg99w" podUID="8e7915fd-e76a-407d-825e-ae095f5d3c56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Oct 30 22:20:14.191188 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:14.191147    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/simulator-w1-2b1e88-kserve-6c579b99f8-q5wpq" podUID="722f10d8-b7a2-4bf7-a274-9669529ef427" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded"
Oct 30 22:20:14.247158 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:14.247118    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/distinct-w2-421583-kserve-69cf5f4d6b-qbgnt" podUID="510ad10d-0554-47c8-9b55-deb359795071" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Oct 30 22:20:14.486246 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:14.486141    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/premium-w6-012ba0-kserve-658654dbdb-d2k27" podUID="085847e5-4f50-419e-984f-b612b104ed90" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:20:14.540185 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:14.540124    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0ad7c5/premium-w4-0ad7c5-kserve-7bbf78cd4-pm6wc" podUID="5c10371e-56e7-42ec-9ef8-0802e15cb2ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:20:14.723066 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:14.719395    4477 patch_prober.go:29] interesting pod/e2e-worker-w3-d52c59-openshift-default-5466968fb7-7m76p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Oct 30 22:20:14.723066 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:14.719454    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-7m76p" podUID="0477d090-6e4c-4599-b4e9-b8e103c67fd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Oct 30 22:20:14.821523 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:14.821432    4477 patch_prober.go:29] interesting pod/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-kvtw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Oct 30 22:20:14.821847 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:14.821819    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-kvtw4" podUID="0a99ab39-789a-4cbf-8db5-5e85bfe485c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Oct 30 22:20:14.849056 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:14.849008    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-421583/distinct-w2-421583-kserve-69cf5f4d6b-qbgnt"
Oct 30 22:20:14.855315 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:14.855291    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0ad7c5/premium-w4-0ad7c5-kserve-7bbf78cd4-pm6wc"
Oct 30 22:20:14.998076 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:14.991769    4477 patch_prober.go:29] interesting pod/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hq9vz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body=
Oct 30 22:20:14.998343 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:14.998311    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hq9vz" podUID="76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused"
Oct 30 22:20:15.124444 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:15.124409    4477 patch_prober.go:29] interesting pod/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-lg99w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Oct 30 22:20:15.124815 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:15.124473    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-lg99w" podUID="8e7915fd-e76a-407d-825e-ae095f5d3c56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Oct 30 22:20:15.721460 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:15.721422    4477 patch_prober.go:29] interesting pod/e2e-worker-w3-d52c59-openshift-default-5466968fb7-7m76p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Oct 30 22:20:15.721753 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:15.721722    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-7m76p" podUID="0477d090-6e4c-4599-b4e9-b8e103c67fd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Oct 30 22:20:16.028111 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:16.027536    4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-kvtw4"
Oct 30 22:20:16.033064 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:16.028764    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-kvtw4"
Oct 30 22:20:16.033064 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:16.033004    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-kvtw4"
Oct 30 22:20:16.120079 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:16.119924    4477 patch_prober.go:29] interesting pod/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-lg99w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Oct 30 22:20:16.120079 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:16.119980    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-lg99w" podUID="8e7915fd-e76a-407d-825e-ae095f5d3c56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Oct 30 22:20:16.149027 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:16.148951    4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hq9vz"
Oct 30 22:20:16.151354 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:16.150854    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hq9vz"
Oct 30 22:20:16.152642 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:16.152623    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hq9vz"
Oct 30 22:20:16.723456 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:16.723426    4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-7m76p"
Oct 30 22:20:16.723816 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:16.723794    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-7m76p"
Oct 30 22:20:16.724389 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:16.724368    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-7m76p"
Oct 30 22:20:17.123810 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:17.123783    4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-lg99w"
Oct 30 22:20:17.124147 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:17.124118    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-lg99w"
Oct 30 22:20:17.124852 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:17.124831    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-lg99w"
Oct 30 22:20:22.290549 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:22.290517    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-pmswk"]
Oct 30 22:20:22.300660 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:22.300618    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-pmswk"
Oct 30 22:20:22.315370 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:22.315349    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-09d0d5-openshift-default-dockercfg-x6h8g\""
Oct 30 22:20:22.328741 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:22.328714    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-pmswk"]
Oct 30 22:20:22.414897 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:22.414869    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7-credential-socket\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-pmswk\" (UID: \"9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-pmswk"
Oct 30 22:20:22.415069 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:22.414913    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7-workload-certs\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-pmswk\" (UID: \"9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-pmswk"
Oct 30 22:20:22.415069 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:22.414931    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7-istiod-ca-cert\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-pmswk\" (UID: \"9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-pmswk"
Oct 30 22:20:22.415216 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:22.415184    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7-istio-data\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-pmswk\" (UID: \"9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-pmswk"
Oct 30 22:20:22.415280 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:22.415239    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7-istio-envoy\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-pmswk\" (UID: \"9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-pmswk"
Oct 30 22:20:22.415336 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:22.415310    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7-istio-token\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-pmswk\" (UID: \"9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-pmswk"
Oct 30 22:20:22.415386 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:22.415361    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7-workload-socket\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-pmswk\" (UID: \"9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-pmswk"
Oct 30 22:20:22.415421 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:22.415400    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89skf\" (UniqueName: \"kubernetes.io/projected/9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7-kube-api-access-89skf\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-pmswk\" (UID: \"9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-pmswk"
Oct 30 22:20:22.415469 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:22.415453    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7-istio-podinfo\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-pmswk\" (UID: \"9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-pmswk"
Oct 30 22:20:22.516344 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:22.516313    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7-istio-podinfo\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-pmswk\" (UID: \"9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-pmswk"
Oct 30 22:20:22.516500 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:22.516378    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7-credential-socket\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-pmswk\" (UID: \"9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-pmswk"
Oct 30 22:20:22.516500 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:22.516406    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7-workload-certs\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-pmswk\" (UID: \"9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-pmswk"
Oct 30 22:20:22.516500 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:22.516421    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7-istiod-ca-cert\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-pmswk\" (UID: \"9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-pmswk"
Oct 30 22:20:22.516671 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:22.516532    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7-istio-data\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-pmswk\" (UID: \"9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-pmswk"
Oct 30 22:20:22.516671 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:22.516577    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7-istio-envoy\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-pmswk\" (UID: \"9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-pmswk"
Oct 30 22:20:22.516671 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:22.516615    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7-istio-token\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-pmswk\" (UID: \"9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-pmswk"
Oct 30 22:20:22.516671 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:22.516640    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7-workload-socket\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-pmswk\" (UID: \"9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-pmswk"
Oct 30 22:20:22.516671 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:22.516663    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89skf\" (UniqueName: \"kubernetes.io/projected/9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7-kube-api-access-89skf\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-pmswk\" (UID: \"9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-pmswk"
Oct 30 22:20:22.517157 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:22.517127    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7-workload-certs\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-pmswk\" (UID: \"9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-pmswk"
Oct 30 22:20:22.517288 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:22.517157    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7-workload-socket\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-pmswk\" (UID: \"9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-pmswk"
Oct 30 22:20:22.517288 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:22.517184    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7-istio-data\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-pmswk\" (UID: \"9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-pmswk"
Oct 30 22:20:22.517288 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:22.517221    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7-credential-socket\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-pmswk\" (UID: \"9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-pmswk"
Oct 30 22:20:22.517602 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:22.517578    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7-istiod-ca-cert\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-pmswk\" (UID: \"9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-pmswk"
Oct 30 22:20:22.518841 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:22.518813    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7-istio-envoy\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-pmswk\" (UID: \"9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-pmswk"
Oct 30 22:20:22.519009 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:22.518990    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7-istio-podinfo\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-pmswk\" (UID: \"9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-pmswk"
Oct 30 22:20:22.537004 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:22.536979    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89skf\" (UniqueName: \"kubernetes.io/projected/9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7-kube-api-access-89skf\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-pmswk\" (UID: \"9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-pmswk"
Oct 30 22:20:22.537461 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:22.537440    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7-istio-token\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-pmswk\" (UID: \"9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-pmswk"
Oct 30 22:20:22.618196 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:22.618121    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-pmswk"
Oct 30 22:20:22.935598 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:22.935565    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-pmswk"]
Oct 30 22:20:22.939413 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:20:22.939389    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c1776ee_cba2_46bd_b9ed_1cf9fd5903d7.slice/crio-09186d3518dbc9cb555def445fb742dd0b64f68d6ef06d3d871d43690abd1825 WatchSource:0}: Error finding container 09186d3518dbc9cb555def445fb742dd0b64f68d6ef06d3d871d43690abd1825: Status 404 returned error can't find the container with id 09186d3518dbc9cb555def445fb742dd0b64f68d6ef06d3d871d43690abd1825
Oct 30 22:20:22.942347 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:22.942313    4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:20:22.942434 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:22.942381    4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:20:22.942434 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:22.942409    4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:20:23.029543 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:23.029519    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-cjkc9"]
Oct 30 22:20:23.033646 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:23.033626    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-cjkc9"
Oct 30 22:20:23.058586 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:23.058558    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-cjkc9"]
Oct 30 22:20:23.122732 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:23.122698    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-2b1e88-openshift-default-dockercfg-ljrwv\""
Oct 30 22:20:23.124214 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:23.124190    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8a3aba3-7d06-4fd5-8106-2d02250f1da2-workload-socket\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-cjkc9\" (UID: \"e8a3aba3-7d06-4fd5-8106-2d02250f1da2\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-cjkc9"
Oct 30 22:20:23.124285 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:23.124251    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8a3aba3-7d06-4fd5-8106-2d02250f1da2-credential-socket\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-cjkc9\" (UID: \"e8a3aba3-7d06-4fd5-8106-2d02250f1da2\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-cjkc9"
Oct 30 22:20:23.124345 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:23.124314    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8a3aba3-7d06-4fd5-8106-2d02250f1da2-istio-podinfo\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-cjkc9\" (UID: \"e8a3aba3-7d06-4fd5-8106-2d02250f1da2\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-cjkc9"
Oct 30 22:20:23.124432 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:23.124416    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8a3aba3-7d06-4fd5-8106-2d02250f1da2-istiod-ca-cert\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-cjkc9\" (UID: \"e8a3aba3-7d06-4fd5-8106-2d02250f1da2\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-cjkc9"
Oct 30 22:20:23.124472 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:23.124452    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqr4w\" (UniqueName: \"kubernetes.io/projected/e8a3aba3-7d06-4fd5-8106-2d02250f1da2-kube-api-access-dqr4w\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-cjkc9\" (UID: \"e8a3aba3-7d06-4fd5-8106-2d02250f1da2\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-cjkc9"
Oct 30 22:20:23.124525 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:23.124487    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8a3aba3-7d06-4fd5-8106-2d02250f1da2-istio-data\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-cjkc9\" (UID: \"e8a3aba3-7d06-4fd5-8106-2d02250f1da2\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-cjkc9"
Oct 30 22:20:23.124568 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:23.124534    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8a3aba3-7d06-4fd5-8106-2d02250f1da2-istio-token\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-cjkc9\" (UID: \"e8a3aba3-7d06-4fd5-8106-2d02250f1da2\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-cjkc9"
Oct 30 22:20:23.124613 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:23.124601    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8a3aba3-7d06-4fd5-8106-2d02250f1da2-workload-certs\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-cjkc9\" (UID: \"e8a3aba3-7d06-4fd5-8106-2d02250f1da2\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-cjkc9"
Oct 30 22:20:23.124650 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:23.124618    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8a3aba3-7d06-4fd5-8106-2d02250f1da2-istio-envoy\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-cjkc9\" (UID: \"e8a3aba3-7d06-4fd5-8106-2d02250f1da2\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-cjkc9"
Oct 30 22:20:23.225468 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:23.225386    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8a3aba3-7d06-4fd5-8106-2d02250f1da2-credential-socket\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-cjkc9\" (UID: \"e8a3aba3-7d06-4fd5-8106-2d02250f1da2\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-cjkc9"
Oct 30 22:20:23.225468 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:23.225429    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8a3aba3-7d06-4fd5-8106-2d02250f1da2-istio-podinfo\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-cjkc9\" (UID: \"e8a3aba3-7d06-4fd5-8106-2d02250f1da2\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-cjkc9"
Oct 30 22:20:23.225650 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:23.225507    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8a3aba3-7d06-4fd5-8106-2d02250f1da2-istiod-ca-cert\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-cjkc9\" (UID: \"e8a3aba3-7d06-4fd5-8106-2d02250f1da2\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-cjkc9"
Oct 30 22:20:23.225650 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:23.225569    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqr4w\" (UniqueName: \"kubernetes.io/projected/e8a3aba3-7d06-4fd5-8106-2d02250f1da2-kube-api-access-dqr4w\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-cjkc9\" (UID: \"e8a3aba3-7d06-4fd5-8106-2d02250f1da2\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-cjkc9"
Oct 30 22:20:23.225650 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:23.225630    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8a3aba3-7d06-4fd5-8106-2d02250f1da2-istio-data\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-cjkc9\" (UID: \"e8a3aba3-7d06-4fd5-8106-2d02250f1da2\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-cjkc9"
Oct 30 22:20:23.225749 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:23.225665    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8a3aba3-7d06-4fd5-8106-2d02250f1da2-istio-token\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-cjkc9\" (UID: \"e8a3aba3-7d06-4fd5-8106-2d02250f1da2\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-cjkc9"
Oct 30 22:20:23.225749 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:23.225718    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8a3aba3-7d06-4fd5-8106-2d02250f1da2-workload-certs\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-cjkc9\" (UID: \"e8a3aba3-7d06-4fd5-8106-2d02250f1da2\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-cjkc9"
Oct 30 22:20:23.225749 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:23.225741    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8a3aba3-7d06-4fd5-8106-2d02250f1da2-istio-envoy\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-cjkc9\" (UID: \"e8a3aba3-7d06-4fd5-8106-2d02250f1da2\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-cjkc9"
Oct 30 22:20:23.225917 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:23.225891    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8a3aba3-7d06-4fd5-8106-2d02250f1da2-workload-socket\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-cjkc9\" (UID: \"e8a3aba3-7d06-4fd5-8106-2d02250f1da2\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-cjkc9"
Oct 30 22:20:23.226227 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:23.226194    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8a3aba3-7d06-4fd5-8106-2d02250f1da2-workload-certs\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-cjkc9\" (UID: \"e8a3aba3-7d06-4fd5-8106-2d02250f1da2\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-cjkc9"
Oct 30 22:20:23.226337 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:23.226276    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8a3aba3-7d06-4fd5-8106-2d02250f1da2-credential-socket\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-cjkc9\" (UID: \"e8a3aba3-7d06-4fd5-8106-2d02250f1da2\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-cjkc9"
Oct 30 22:20:23.226727 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:23.226452    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8a3aba3-7d06-4fd5-8106-2d02250f1da2-workload-socket\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-cjkc9\" (UID: \"e8a3aba3-7d06-4fd5-8106-2d02250f1da2\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-cjkc9"
Oct 30 22:20:23.226727 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:23.226605    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8a3aba3-7d06-4fd5-8106-2d02250f1da2-istio-data\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-cjkc9\" (UID: \"e8a3aba3-7d06-4fd5-8106-2d02250f1da2\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-cjkc9"
Oct 30 22:20:23.227137 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:23.227108    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8a3aba3-7d06-4fd5-8106-2d02250f1da2-istiod-ca-cert\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-cjkc9\" (UID: \"e8a3aba3-7d06-4fd5-8106-2d02250f1da2\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-cjkc9"
Oct 30 22:20:23.228535 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:23.228511    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8a3aba3-7d06-4fd5-8106-2d02250f1da2-istio-envoy\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-cjkc9\" (UID: \"e8a3aba3-7d06-4fd5-8106-2d02250f1da2\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-cjkc9"
Oct 30 22:20:23.228646 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:23.228626    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8a3aba3-7d06-4fd5-8106-2d02250f1da2-istio-podinfo\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-cjkc9\" (UID: \"e8a3aba3-7d06-4fd5-8106-2d02250f1da2\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-cjkc9"
Oct 30 22:20:23.244226 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:23.244200    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8a3aba3-7d06-4fd5-8106-2d02250f1da2-istio-token\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-cjkc9\" (UID: \"e8a3aba3-7d06-4fd5-8106-2d02250f1da2\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-cjkc9"
Oct 30 22:20:23.245229 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:23.245209    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqr4w\" (UniqueName: \"kubernetes.io/projected/e8a3aba3-7d06-4fd5-8106-2d02250f1da2-kube-api-access-dqr4w\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-cjkc9\" (UID: \"e8a3aba3-7d06-4fd5-8106-2d02250f1da2\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-cjkc9"
Oct 30 22:20:23.350354 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:23.350320    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-cjkc9"
Oct 30 22:20:23.605262 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:23.605168    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-pmswk" event={"ID":"9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7","Type":"ContainerStarted","Data":"ed35cf609bf857a95d6417e89e7792109358a7da9992f72952de20867bf7ce4c"}
Oct 30 22:20:23.605262 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:23.605225    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-pmswk" event={"ID":"9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7","Type":"ContainerStarted","Data":"09186d3518dbc9cb555def445fb742dd0b64f68d6ef06d3d871d43690abd1825"}
Oct 30 22:20:23.618843 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:23.618811    4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-pmswk"
Oct 30 22:20:23.646194 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:23.646137    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-pmswk" podStartSLOduration=1.646123819 podStartE2EDuration="1.646123819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:20:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:20:23.6443738 +0000 UTC m=+807.570300043" watchObservedRunningTime="2026-10-30 22:20:23.646123819 +0000 UTC m=+807.572050025"
Oct 30 22:20:23.695731 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:23.695636    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-cjkc9"]
Oct 30 22:20:23.700349 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:20:23.700318    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8a3aba3_7d06_4fd5_8106_2d02250f1da2.slice/crio-7b56f5622e6920d5107afe8f3a0d8c883d59419da5d18ee840fdaa36cbe69eeb WatchSource:0}: Error finding container 7b56f5622e6920d5107afe8f3a0d8c883d59419da5d18ee840fdaa36cbe69eeb: Status 404 returned error can't find the container with id 7b56f5622e6920d5107afe8f3a0d8c883d59419da5d18ee840fdaa36cbe69eeb
Oct 30 22:20:23.702817 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:23.702779    4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:20:23.702925 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:23.702860    4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:20:23.702925 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:23.702906    4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:20:24.623364 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:24.623320    4477 patch_prober.go:29] interesting pod/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-pmswk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Oct 30 22:20:24.624030 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:24.623920    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-pmswk" podUID="9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Oct 30 22:20:24.627323 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:24.627237    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-cjkc9" event={"ID":"e8a3aba3-7d06-4fd5-8106-2d02250f1da2","Type":"ContainerStarted","Data":"019a986e536c9e2adbd8f4c7a9976538e38f3215dbb98967f7b5f12eb6167d2a"}
Oct 30 22:20:24.627323 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:24.627292    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-cjkc9" event={"ID":"e8a3aba3-7d06-4fd5-8106-2d02250f1da2","Type":"ContainerStarted","Data":"7b56f5622e6920d5107afe8f3a0d8c883d59419da5d18ee840fdaa36cbe69eeb"}
Oct 30 22:20:24.725120 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:24.723234    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-cjkc9" podStartSLOduration=1.7232151089999999 podStartE2EDuration="1.723215109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:20:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:20:24.681814565 +0000 UTC m=+808.607740772" watchObservedRunningTime="2026-10-30 22:20:24.723215109 +0000 UTC m=+808.649141316"
Oct 30 22:20:25.350629 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:25.350602    4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-cjkc9"
Oct 30 22:20:25.351695 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:25.351670    4477 patch_prober.go:29] interesting pod/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-cjkc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Oct 30 22:20:25.351808 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:25.351714    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-cjkc9" podUID="e8a3aba3-7d06-4fd5-8106-2d02250f1da2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Oct 30 22:20:25.619833 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:25.619616    4477 patch_prober.go:29] interesting pod/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-pmswk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Oct 30 22:20:25.619833 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:25.619714    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-pmswk" podUID="9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Oct 30 22:20:26.265113 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:20:26.265077    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 22:20:26.351512 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:26.351472    4477 patch_prober.go:29] interesting pod/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-cjkc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Oct 30 22:20:26.351676 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:26.351533    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-cjkc9" podUID="e8a3aba3-7d06-4fd5-8106-2d02250f1da2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Oct 30 22:20:26.619545 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:26.619470    4477 patch_prober.go:29] interesting pod/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-pmswk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Oct 30 22:20:26.619751 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:26.619727    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-pmswk" podUID="9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Oct 30 22:20:27.351342 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:27.351310    4477 patch_prober.go:29] interesting pod/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-cjkc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Oct 30 22:20:27.351796 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:27.351770    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-cjkc9" podUID="e8a3aba3-7d06-4fd5-8106-2d02250f1da2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Oct 30 22:20:27.621165 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:27.620971    4477 patch_prober.go:29] interesting pod/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-pmswk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Oct 30 22:20:27.621165 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:27.621071    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-pmswk" podUID="9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Oct 30 22:20:28.351238 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:28.351182    4477 patch_prober.go:29] interesting pod/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-cjkc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Oct 30 22:20:28.351419 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:28.351253    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-cjkc9" podUID="e8a3aba3-7d06-4fd5-8106-2d02250f1da2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Oct 30 22:20:28.619495 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:28.619402    4477 patch_prober.go:29] interesting pod/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-pmswk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Oct 30 22:20:28.619852 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:28.619823    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-pmswk" podUID="9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Oct 30 22:20:29.350786 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:29.350751    4477 patch_prober.go:29] interesting pod/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-cjkc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Oct 30 22:20:29.350985 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:29.350803    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-cjkc9" podUID="e8a3aba3-7d06-4fd5-8106-2d02250f1da2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Oct 30 22:20:29.805400 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:29.805368    4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-pmswk"
Oct 30 22:20:29.805846 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:29.805825    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-pmswk"
Oct 30 22:20:29.807176 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:29.807149    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-pmswk"
Oct 30 22:20:30.449009 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:30.448975    4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-cjkc9"
Oct 30 22:20:30.449383 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:30.449362    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-cjkc9"
Oct 30 22:20:30.451308 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:30.451285    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-cjkc9"
Oct 30 22:20:30.916897 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:30.916865    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6bz92"]
Oct 30 22:20:55.473117 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:55.473086    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-012ba0/distinct-2-w6-012ba0-kserve-78dbcd8c46-hqdfm"]
Oct 30 22:20:55.477185 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:55.477160    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-012ba0/distinct-2-w6-012ba0-kserve-78dbcd8c46-hqdfm"
Oct 30 22:20:55.500557 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:55.500532    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-012ba0\"/\"distinct-2-w6-012ba0-kserve-self-signed-certs\""
Oct 30 22:20:55.503499 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:55.503478    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-012ba0/distinct-2-w6-012ba0-kserve-78dbcd8c46-hqdfm"]
Oct 30 22:20:55.557943 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:55.557915    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8326b014-989b-4722-befb-6fa8ea60688b-home\") pod \"distinct-2-w6-012ba0-kserve-78dbcd8c46-hqdfm\" (UID: \"8326b014-989b-4722-befb-6fa8ea60688b\") " pod="e2e-models-e2e-worker-w6-012ba0/distinct-2-w6-012ba0-kserve-78dbcd8c46-hqdfm"
Oct 30 22:20:55.558093 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:55.558064    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7xr2\" (UniqueName: \"kubernetes.io/projected/8326b014-989b-4722-befb-6fa8ea60688b-kube-api-access-g7xr2\") pod \"distinct-2-w6-012ba0-kserve-78dbcd8c46-hqdfm\" (UID: \"8326b014-989b-4722-befb-6fa8ea60688b\") " pod="e2e-models-e2e-worker-w6-012ba0/distinct-2-w6-012ba0-kserve-78dbcd8c46-hqdfm"
Oct 30 22:20:55.558204 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:55.558186    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8326b014-989b-4722-befb-6fa8ea60688b-tmp-dir\") pod \"distinct-2-w6-012ba0-kserve-78dbcd8c46-hqdfm\" (UID: \"8326b014-989b-4722-befb-6fa8ea60688b\") " pod="e2e-models-e2e-worker-w6-012ba0/distinct-2-w6-012ba0-kserve-78dbcd8c46-hqdfm"
Oct 30 22:20:55.558253 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:55.558239    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8326b014-989b-4722-befb-6fa8ea60688b-model-cache\") pod \"distinct-2-w6-012ba0-kserve-78dbcd8c46-hqdfm\" (UID: \"8326b014-989b-4722-befb-6fa8ea60688b\") " pod="e2e-models-e2e-worker-w6-012ba0/distinct-2-w6-012ba0-kserve-78dbcd8c46-hqdfm"
Oct 30 22:20:55.558374 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:55.558350    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8326b014-989b-4722-befb-6fa8ea60688b-tls-certs\") pod \"distinct-2-w6-012ba0-kserve-78dbcd8c46-hqdfm\" (UID: \"8326b014-989b-4722-befb-6fa8ea60688b\") " pod="e2e-models-e2e-worker-w6-012ba0/distinct-2-w6-012ba0-kserve-78dbcd8c46-hqdfm"
Oct 30 22:20:55.558423 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:55.558408    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8326b014-989b-4722-befb-6fa8ea60688b-dshm\") pod \"distinct-2-w6-012ba0-kserve-78dbcd8c46-hqdfm\" (UID: \"8326b014-989b-4722-befb-6fa8ea60688b\") " pod="e2e-models-e2e-worker-w6-012ba0/distinct-2-w6-012ba0-kserve-78dbcd8c46-hqdfm"
Oct 30 22:20:55.660061 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:55.659652    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8326b014-989b-4722-befb-6fa8ea60688b-home\") pod \"distinct-2-w6-012ba0-kserve-78dbcd8c46-hqdfm\" (UID: \"8326b014-989b-4722-befb-6fa8ea60688b\") " pod="e2e-models-e2e-worker-w6-012ba0/distinct-2-w6-012ba0-kserve-78dbcd8c46-hqdfm"
Oct 30 22:20:55.660061 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:55.659751    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7xr2\" (UniqueName: \"kubernetes.io/projected/8326b014-989b-4722-befb-6fa8ea60688b-kube-api-access-g7xr2\") pod \"distinct-2-w6-012ba0-kserve-78dbcd8c46-hqdfm\" (UID: \"8326b014-989b-4722-befb-6fa8ea60688b\") " pod="e2e-models-e2e-worker-w6-012ba0/distinct-2-w6-012ba0-kserve-78dbcd8c46-hqdfm"
Oct 30 22:20:55.660061 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:55.659821    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8326b014-989b-4722-befb-6fa8ea60688b-tmp-dir\") pod \"distinct-2-w6-012ba0-kserve-78dbcd8c46-hqdfm\" (UID: \"8326b014-989b-4722-befb-6fa8ea60688b\") " pod="e2e-models-e2e-worker-w6-012ba0/distinct-2-w6-012ba0-kserve-78dbcd8c46-hqdfm"
Oct 30 22:20:55.660061 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:55.659869    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8326b014-989b-4722-befb-6fa8ea60688b-model-cache\") pod \"distinct-2-w6-012ba0-kserve-78dbcd8c46-hqdfm\" (UID: \"8326b014-989b-4722-befb-6fa8ea60688b\") " pod="e2e-models-e2e-worker-w6-012ba0/distinct-2-w6-012ba0-kserve-78dbcd8c46-hqdfm"
Oct 30 22:20:55.660061 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:55.659911    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8326b014-989b-4722-befb-6fa8ea60688b-tls-certs\") pod \"distinct-2-w6-012ba0-kserve-78dbcd8c46-hqdfm\" (UID: \"8326b014-989b-4722-befb-6fa8ea60688b\") " pod="e2e-models-e2e-worker-w6-012ba0/distinct-2-w6-012ba0-kserve-78dbcd8c46-hqdfm"
Oct 30 22:20:55.660451 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:55.660125    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8326b014-989b-4722-befb-6fa8ea60688b-home\") pod \"distinct-2-w6-012ba0-kserve-78dbcd8c46-hqdfm\" (UID: \"8326b014-989b-4722-befb-6fa8ea60688b\") " pod="e2e-models-e2e-worker-w6-012ba0/distinct-2-w6-012ba0-kserve-78dbcd8c46-hqdfm"
Oct 30 22:20:55.660451 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:55.660133    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8326b014-989b-4722-befb-6fa8ea60688b-dshm\") pod \"distinct-2-w6-012ba0-kserve-78dbcd8c46-hqdfm\" (UID: \"8326b014-989b-4722-befb-6fa8ea60688b\") " pod="e2e-models-e2e-worker-w6-012ba0/distinct-2-w6-012ba0-kserve-78dbcd8c46-hqdfm"
Oct 30 22:20:55.660877 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:55.660611    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8326b014-989b-4722-befb-6fa8ea60688b-tmp-dir\") pod \"distinct-2-w6-012ba0-kserve-78dbcd8c46-hqdfm\" (UID: \"8326b014-989b-4722-befb-6fa8ea60688b\") " pod="e2e-models-e2e-worker-w6-012ba0/distinct-2-w6-012ba0-kserve-78dbcd8c46-hqdfm"
Oct 30 22:20:55.660991 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:55.660858    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8326b014-989b-4722-befb-6fa8ea60688b-model-cache\") pod \"distinct-2-w6-012ba0-kserve-78dbcd8c46-hqdfm\" (UID: \"8326b014-989b-4722-befb-6fa8ea60688b\") " pod="e2e-models-e2e-worker-w6-012ba0/distinct-2-w6-012ba0-kserve-78dbcd8c46-hqdfm"
Oct 30 22:20:55.663335 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:55.663310    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8326b014-989b-4722-befb-6fa8ea60688b-dshm\") pod \"distinct-2-w6-012ba0-kserve-78dbcd8c46-hqdfm\" (UID: \"8326b014-989b-4722-befb-6fa8ea60688b\") " pod="e2e-models-e2e-worker-w6-012ba0/distinct-2-w6-012ba0-kserve-78dbcd8c46-hqdfm"
Oct 30 22:20:55.664132 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:55.664108    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8326b014-989b-4722-befb-6fa8ea60688b-tls-certs\") pod \"distinct-2-w6-012ba0-kserve-78dbcd8c46-hqdfm\" (UID: \"8326b014-989b-4722-befb-6fa8ea60688b\") " pod="e2e-models-e2e-worker-w6-012ba0/distinct-2-w6-012ba0-kserve-78dbcd8c46-hqdfm"
Oct 30 22:20:55.686438 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:55.686406    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7xr2\" (UniqueName: \"kubernetes.io/projected/8326b014-989b-4722-befb-6fa8ea60688b-kube-api-access-g7xr2\") pod \"distinct-2-w6-012ba0-kserve-78dbcd8c46-hqdfm\" (UID: \"8326b014-989b-4722-befb-6fa8ea60688b\") " pod="e2e-models-e2e-worker-w6-012ba0/distinct-2-w6-012ba0-kserve-78dbcd8c46-hqdfm"
Oct 30 22:20:55.792519 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:55.792428    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-012ba0/distinct-2-w6-012ba0-kserve-78dbcd8c46-hqdfm"
Oct 30 22:20:56.094140 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:56.094030    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-012ba0/distinct-2-w6-012ba0-kserve-78dbcd8c46-hqdfm"]
Oct 30 22:20:56.098029 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:20:56.097998    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8326b014_989b_4722_befb_6fa8ea60688b.slice/crio-19b77357627087ee3e9bea6fe79f2df5b62b276843970bff85ef10cb793be06d WatchSource:0}: Error finding container 19b77357627087ee3e9bea6fe79f2df5b62b276843970bff85ef10cb793be06d: Status 404 returned error can't find the container with id 19b77357627087ee3e9bea6fe79f2df5b62b276843970bff85ef10cb793be06d
Oct 30 22:20:56.265635 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:20:56.265600    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 22:20:56.861626 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:56.861595    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-012ba0/distinct-2-w6-012ba0-kserve-78dbcd8c46-hqdfm" event={"ID":"8326b014-989b-4722-befb-6fa8ea60688b","Type":"ContainerStarted","Data":"8755ce6e726a15081433d3a94bac9ad168a0241c8a88446a91721ed2bd38588c"}
Oct 30 22:20:56.862090 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:56.861634    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-012ba0/distinct-2-w6-012ba0-kserve-78dbcd8c46-hqdfm" event={"ID":"8326b014-989b-4722-befb-6fa8ea60688b","Type":"ContainerStarted","Data":"19b77357627087ee3e9bea6fe79f2df5b62b276843970bff85ef10cb793be06d"}
Oct 30 22:20:56.901858 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:56.901811    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-012ba0/distinct-2-w6-012ba0-kserve-78dbcd8c46-hqdfm" podStartSLOduration=1.901797278 podStartE2EDuration="1.901797278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:20:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:20:56.898380479 +0000 UTC m=+840.824306684" watchObservedRunningTime="2026-10-30 22:20:56.901797278 +0000 UTC m=+840.827723483"
Oct 30 22:20:57.793132 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:57.793099    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-012ba0/distinct-2-w6-012ba0-kserve-78dbcd8c46-hqdfm"
Oct 30 22:20:58.614962 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:58.614925    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-0ad7c5/distinct-2-w4-0ad7c5-kserve-77c5467fcb-skspk"]
Oct 30 22:20:58.621371 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:58.621346    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-2-w4-0ad7c5-kserve-77c5467fcb-skspk"
Oct 30 22:20:58.645349 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:58.645305    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-0ad7c5\"/\"distinct-2-w4-0ad7c5-kserve-self-signed-certs\""
Oct 30 22:20:58.649737 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:58.649712    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-0ad7c5/distinct-2-w4-0ad7c5-kserve-77c5467fcb-skspk"]
Oct 30 22:20:58.800793 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:58.800758    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/430bd879-2a89-4070-bb1b-9b9958a261b9-dshm\") pod \"distinct-2-w4-0ad7c5-kserve-77c5467fcb-skspk\" (UID: \"430bd879-2a89-4070-bb1b-9b9958a261b9\") " pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-2-w4-0ad7c5-kserve-77c5467fcb-skspk"
Oct 30 22:20:58.800965 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:58.800828    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgjh8\" (UniqueName: \"kubernetes.io/projected/430bd879-2a89-4070-bb1b-9b9958a261b9-kube-api-access-dgjh8\") pod \"distinct-2-w4-0ad7c5-kserve-77c5467fcb-skspk\" (UID: \"430bd879-2a89-4070-bb1b-9b9958a261b9\") " pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-2-w4-0ad7c5-kserve-77c5467fcb-skspk"
Oct 30 22:20:58.800965 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:58.800898    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/430bd879-2a89-4070-bb1b-9b9958a261b9-tmp-dir\") pod \"distinct-2-w4-0ad7c5-kserve-77c5467fcb-skspk\" (UID: \"430bd879-2a89-4070-bb1b-9b9958a261b9\") " pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-2-w4-0ad7c5-kserve-77c5467fcb-skspk"
Oct 30 22:20:58.801091 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:58.800998    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/430bd879-2a89-4070-bb1b-9b9958a261b9-model-cache\") pod \"distinct-2-w4-0ad7c5-kserve-77c5467fcb-skspk\" (UID: \"430bd879-2a89-4070-bb1b-9b9958a261b9\") " pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-2-w4-0ad7c5-kserve-77c5467fcb-skspk"
Oct 30 22:20:58.801091 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:58.801029    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/430bd879-2a89-4070-bb1b-9b9958a261b9-tls-certs\") pod \"distinct-2-w4-0ad7c5-kserve-77c5467fcb-skspk\" (UID: \"430bd879-2a89-4070-bb1b-9b9958a261b9\") " pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-2-w4-0ad7c5-kserve-77c5467fcb-skspk"
Oct 30 22:20:58.801237 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:58.801215    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/430bd879-2a89-4070-bb1b-9b9958a261b9-home\") pod \"distinct-2-w4-0ad7c5-kserve-77c5467fcb-skspk\" (UID: \"430bd879-2a89-4070-bb1b-9b9958a261b9\") " pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-2-w4-0ad7c5-kserve-77c5467fcb-skspk"
Oct 30 22:20:58.904075 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:58.902973    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/430bd879-2a89-4070-bb1b-9b9958a261b9-model-cache\") pod \"distinct-2-w4-0ad7c5-kserve-77c5467fcb-skspk\" (UID: \"430bd879-2a89-4070-bb1b-9b9958a261b9\") " pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-2-w4-0ad7c5-kserve-77c5467fcb-skspk"
Oct 30 22:20:58.904075 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:58.903022    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/430bd879-2a89-4070-bb1b-9b9958a261b9-tls-certs\") pod \"distinct-2-w4-0ad7c5-kserve-77c5467fcb-skspk\" (UID: \"430bd879-2a89-4070-bb1b-9b9958a261b9\") " pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-2-w4-0ad7c5-kserve-77c5467fcb-skspk"
Oct 30 22:20:58.904075 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:58.903199    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/430bd879-2a89-4070-bb1b-9b9958a261b9-home\") pod \"distinct-2-w4-0ad7c5-kserve-77c5467fcb-skspk\" (UID: \"430bd879-2a89-4070-bb1b-9b9958a261b9\") " pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-2-w4-0ad7c5-kserve-77c5467fcb-skspk"
Oct 30 22:20:58.904075 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:58.903313    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/430bd879-2a89-4070-bb1b-9b9958a261b9-dshm\") pod \"distinct-2-w4-0ad7c5-kserve-77c5467fcb-skspk\" (UID: \"430bd879-2a89-4070-bb1b-9b9958a261b9\") " pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-2-w4-0ad7c5-kserve-77c5467fcb-skspk"
Oct 30 22:20:58.904075 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:58.903367    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgjh8\" (UniqueName: \"kubernetes.io/projected/430bd879-2a89-4070-bb1b-9b9958a261b9-kube-api-access-dgjh8\") pod \"distinct-2-w4-0ad7c5-kserve-77c5467fcb-skspk\" (UID: \"430bd879-2a89-4070-bb1b-9b9958a261b9\") " pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-2-w4-0ad7c5-kserve-77c5467fcb-skspk"
Oct 30 22:20:58.904075 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:58.903402    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/430bd879-2a89-4070-bb1b-9b9958a261b9-tmp-dir\") pod \"distinct-2-w4-0ad7c5-kserve-77c5467fcb-skspk\" (UID: \"430bd879-2a89-4070-bb1b-9b9958a261b9\") " pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-2-w4-0ad7c5-kserve-77c5467fcb-skspk"
Oct 30 22:20:58.904075 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:58.903471    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/430bd879-2a89-4070-bb1b-9b9958a261b9-model-cache\") pod \"distinct-2-w4-0ad7c5-kserve-77c5467fcb-skspk\" (UID: \"430bd879-2a89-4070-bb1b-9b9958a261b9\") " pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-2-w4-0ad7c5-kserve-77c5467fcb-skspk"
Oct 30 22:20:58.904075 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:58.903732    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/430bd879-2a89-4070-bb1b-9b9958a261b9-home\") pod \"distinct-2-w4-0ad7c5-kserve-77c5467fcb-skspk\" (UID: \"430bd879-2a89-4070-bb1b-9b9958a261b9\") " pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-2-w4-0ad7c5-kserve-77c5467fcb-skspk"
Oct 30 22:20:58.904075 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:58.903815    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/430bd879-2a89-4070-bb1b-9b9958a261b9-tmp-dir\") pod \"distinct-2-w4-0ad7c5-kserve-77c5467fcb-skspk\" (UID: \"430bd879-2a89-4070-bb1b-9b9958a261b9\") " pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-2-w4-0ad7c5-kserve-77c5467fcb-skspk"
Oct 30 22:20:58.907116 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:58.907060    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/430bd879-2a89-4070-bb1b-9b9958a261b9-dshm\") pod \"distinct-2-w4-0ad7c5-kserve-77c5467fcb-skspk\" (UID: \"430bd879-2a89-4070-bb1b-9b9958a261b9\") " pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-2-w4-0ad7c5-kserve-77c5467fcb-skspk"
Oct 30 22:20:58.907577 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:58.907555    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/430bd879-2a89-4070-bb1b-9b9958a261b9-tls-certs\") pod \"distinct-2-w4-0ad7c5-kserve-77c5467fcb-skspk\" (UID: \"430bd879-2a89-4070-bb1b-9b9958a261b9\") " pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-2-w4-0ad7c5-kserve-77c5467fcb-skspk"
Oct 30 22:20:58.922301 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:58.922275    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgjh8\" (UniqueName: \"kubernetes.io/projected/430bd879-2a89-4070-bb1b-9b9958a261b9-kube-api-access-dgjh8\") pod \"distinct-2-w4-0ad7c5-kserve-77c5467fcb-skspk\" (UID: \"430bd879-2a89-4070-bb1b-9b9958a261b9\") " pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-2-w4-0ad7c5-kserve-77c5467fcb-skspk"
Oct 30 22:20:58.935712 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:58.935690    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-2-w4-0ad7c5-kserve-77c5467fcb-skspk"
Oct 30 22:20:59.243932 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:59.243855    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-0ad7c5/distinct-2-w4-0ad7c5-kserve-77c5467fcb-skspk"]
Oct 30 22:20:59.247439 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:20:59.247405    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod430bd879_2a89_4070_bb1b_9b9958a261b9.slice/crio-01c1b3f250534bb730ea450723da9a7a5a93e59fd0ce2717374764e2b220b697 WatchSource:0}: Error finding container 01c1b3f250534bb730ea450723da9a7a5a93e59fd0ce2717374764e2b220b697: Status 404 returned error can't find the container with id 01c1b3f250534bb730ea450723da9a7a5a93e59fd0ce2717374764e2b220b697
Oct 30 22:20:59.884114 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:59.884081    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-2-w4-0ad7c5-kserve-77c5467fcb-skspk" event={"ID":"430bd879-2a89-4070-bb1b-9b9958a261b9","Type":"ContainerStarted","Data":"8b0365cdaed18f133a96bf414c525bd03d2117181ce757a4b52a804d337c1d9b"}
Oct 30 22:20:59.884535 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:59.884125    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-2-w4-0ad7c5-kserve-77c5467fcb-skspk" event={"ID":"430bd879-2a89-4070-bb1b-9b9958a261b9","Type":"ContainerStarted","Data":"01c1b3f250534bb730ea450723da9a7a5a93e59fd0ce2717374764e2b220b697"}
Oct 30 22:20:59.919421 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:59.919379    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-2-w4-0ad7c5-kserve-77c5467fcb-skspk" podStartSLOduration=1.919365926 podStartE2EDuration="1.919365926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:20:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:20:59.91879007 +0000 UTC m=+843.844716300" watchObservedRunningTime="2026-10-30 22:20:59.919365926 +0000 UTC m=+843.845292153"
Oct 30 22:20:59.936034 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:20:59.936010    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-2-w4-0ad7c5-kserve-77c5467fcb-skspk"
Oct 30 22:21:05.793118 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:05.793087    4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-012ba0/distinct-2-w6-012ba0-kserve-78dbcd8c46-hqdfm"
Oct 30 22:21:05.805701 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:05.805668    4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-012ba0/distinct-2-w6-012ba0-kserve-78dbcd8c46-hqdfm"
Oct 30 22:21:05.929177 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:05.929148    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-012ba0/distinct-2-w6-012ba0-kserve-78dbcd8c46-hqdfm"
Oct 30 22:21:08.943823 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:08.937682    4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-2-w4-0ad7c5-kserve-77c5467fcb-skspk"
Oct 30 22:21:08.955114 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:08.955090    4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-2-w4-0ad7c5-kserve-77c5467fcb-skspk"
Oct 30 22:21:09.947655 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:09.947629    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-2-w4-0ad7c5-kserve-77c5467fcb-skspk"
Oct 30 22:21:11.665012 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:11.664985    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6bz92"]
Oct 30 22:21:14.112765 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:14.112729    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-09d0d5/distinct-2-w0-09d0d5-kserve-7dcf89b98b-7cjph"]
Oct 30 22:21:14.116796 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:14.116765    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-09d0d5/distinct-2-w0-09d0d5-kserve-7dcf89b98b-7cjph"
Oct 30 22:21:14.142790 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:14.142763    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-09d0d5\"/\"distinct-2-w0-09d0d5-kserve-self-signed-certs\""
Oct 30 22:21:14.142967 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:14.142779    4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-09d0d5\"/\"openshift-service-ca.crt\""
Oct 30 22:21:14.143346 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:14.143328    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-09d0d5\"/\"default-dockercfg-69b2d\""
Oct 30 22:21:14.145967 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:14.145947    4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-09d0d5\"/\"kube-root-ca.crt\""
Oct 30 22:21:14.146097 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:14.146010    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-09d0d5/distinct-2-w0-09d0d5-kserve-7dcf89b98b-7cjph"]
Oct 30 22:21:14.172576 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:14.172548    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8cbd04b4-bb3e-4bbb-a41a-46131c9de051-tls-certs\") pod \"distinct-2-w0-09d0d5-kserve-7dcf89b98b-7cjph\" (UID: \"8cbd04b4-bb3e-4bbb-a41a-46131c9de051\") " pod="e2e-models-e2e-worker-w0-09d0d5/distinct-2-w0-09d0d5-kserve-7dcf89b98b-7cjph"
Oct 30 22:21:14.172691 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:14.172608    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5zq8\" (UniqueName: \"kubernetes.io/projected/8cbd04b4-bb3e-4bbb-a41a-46131c9de051-kube-api-access-h5zq8\") pod \"distinct-2-w0-09d0d5-kserve-7dcf89b98b-7cjph\" (UID: \"8cbd04b4-bb3e-4bbb-a41a-46131c9de051\") " pod="e2e-models-e2e-worker-w0-09d0d5/distinct-2-w0-09d0d5-kserve-7dcf89b98b-7cjph"
Oct 30 22:21:14.172756 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:14.172722    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8cbd04b4-bb3e-4bbb-a41a-46131c9de051-dshm\") pod \"distinct-2-w0-09d0d5-kserve-7dcf89b98b-7cjph\" (UID: \"8cbd04b4-bb3e-4bbb-a41a-46131c9de051\") " pod="e2e-models-e2e-worker-w0-09d0d5/distinct-2-w0-09d0d5-kserve-7dcf89b98b-7cjph"
Oct 30 22:21:14.172810 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:14.172765    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8cbd04b4-bb3e-4bbb-a41a-46131c9de051-tmp-dir\") pod \"distinct-2-w0-09d0d5-kserve-7dcf89b98b-7cjph\" (UID: \"8cbd04b4-bb3e-4bbb-a41a-46131c9de051\") " pod="e2e-models-e2e-worker-w0-09d0d5/distinct-2-w0-09d0d5-kserve-7dcf89b98b-7cjph"
Oct 30 22:21:14.172862 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:14.172846    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8cbd04b4-bb3e-4bbb-a41a-46131c9de051-model-cache\") pod \"distinct-2-w0-09d0d5-kserve-7dcf89b98b-7cjph\" (UID: \"8cbd04b4-bb3e-4bbb-a41a-46131c9de051\") " pod="e2e-models-e2e-worker-w0-09d0d5/distinct-2-w0-09d0d5-kserve-7dcf89b98b-7cjph"
Oct 30 22:21:14.172914 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:14.172875    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8cbd04b4-bb3e-4bbb-a41a-46131c9de051-home\") pod \"distinct-2-w0-09d0d5-kserve-7dcf89b98b-7cjph\" (UID: \"8cbd04b4-bb3e-4bbb-a41a-46131c9de051\") " pod="e2e-models-e2e-worker-w0-09d0d5/distinct-2-w0-09d0d5-kserve-7dcf89b98b-7cjph"
Oct 30 22:21:14.274316 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:14.274282    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8cbd04b4-bb3e-4bbb-a41a-46131c9de051-model-cache\") pod \"distinct-2-w0-09d0d5-kserve-7dcf89b98b-7cjph\" (UID: \"8cbd04b4-bb3e-4bbb-a41a-46131c9de051\") " pod="e2e-models-e2e-worker-w0-09d0d5/distinct-2-w0-09d0d5-kserve-7dcf89b98b-7cjph"
Oct 30 22:21:14.274476 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:14.274338    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8cbd04b4-bb3e-4bbb-a41a-46131c9de051-home\") pod \"distinct-2-w0-09d0d5-kserve-7dcf89b98b-7cjph\" (UID: \"8cbd04b4-bb3e-4bbb-a41a-46131c9de051\") " pod="e2e-models-e2e-worker-w0-09d0d5/distinct-2-w0-09d0d5-kserve-7dcf89b98b-7cjph"
Oct 30 22:21:14.274476 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:14.274406    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8cbd04b4-bb3e-4bbb-a41a-46131c9de051-tls-certs\") pod \"distinct-2-w0-09d0d5-kserve-7dcf89b98b-7cjph\" (UID: \"8cbd04b4-bb3e-4bbb-a41a-46131c9de051\") " pod="e2e-models-e2e-worker-w0-09d0d5/distinct-2-w0-09d0d5-kserve-7dcf89b98b-7cjph"
Oct 30 22:21:14.274594 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:14.274556    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5zq8\" (UniqueName: \"kubernetes.io/projected/8cbd04b4-bb3e-4bbb-a41a-46131c9de051-kube-api-access-h5zq8\") pod \"distinct-2-w0-09d0d5-kserve-7dcf89b98b-7cjph\" (UID: \"8cbd04b4-bb3e-4bbb-a41a-46131c9de051\") " pod="e2e-models-e2e-worker-w0-09d0d5/distinct-2-w0-09d0d5-kserve-7dcf89b98b-7cjph"
Oct 30 22:21:14.274677 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:14.274652    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8cbd04b4-bb3e-4bbb-a41a-46131c9de051-dshm\") pod \"distinct-2-w0-09d0d5-kserve-7dcf89b98b-7cjph\" (UID: \"8cbd04b4-bb3e-4bbb-a41a-46131c9de051\") " pod="e2e-models-e2e-worker-w0-09d0d5/distinct-2-w0-09d0d5-kserve-7dcf89b98b-7cjph"
Oct 30 22:21:14.274742 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:14.274698    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8cbd04b4-bb3e-4bbb-a41a-46131c9de051-tmp-dir\") pod \"distinct-2-w0-09d0d5-kserve-7dcf89b98b-7cjph\" (UID: \"8cbd04b4-bb3e-4bbb-a41a-46131c9de051\") " pod="e2e-models-e2e-worker-w0-09d0d5/distinct-2-w0-09d0d5-kserve-7dcf89b98b-7cjph"
Oct 30 22:21:14.274742 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:14.274700    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8cbd04b4-bb3e-4bbb-a41a-46131c9de051-model-cache\") pod \"distinct-2-w0-09d0d5-kserve-7dcf89b98b-7cjph\" (UID: \"8cbd04b4-bb3e-4bbb-a41a-46131c9de051\") " pod="e2e-models-e2e-worker-w0-09d0d5/distinct-2-w0-09d0d5-kserve-7dcf89b98b-7cjph"
Oct 30 22:21:14.274848 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:14.274756    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8cbd04b4-bb3e-4bbb-a41a-46131c9de051-home\") pod \"distinct-2-w0-09d0d5-kserve-7dcf89b98b-7cjph\" (UID: \"8cbd04b4-bb3e-4bbb-a41a-46131c9de051\") " pod="e2e-models-e2e-worker-w0-09d0d5/distinct-2-w0-09d0d5-kserve-7dcf89b98b-7cjph"
Oct 30 22:21:14.275130 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:14.275109    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8cbd04b4-bb3e-4bbb-a41a-46131c9de051-tmp-dir\") pod \"distinct-2-w0-09d0d5-kserve-7dcf89b98b-7cjph\" (UID: \"8cbd04b4-bb3e-4bbb-a41a-46131c9de051\") " pod="e2e-models-e2e-worker-w0-09d0d5/distinct-2-w0-09d0d5-kserve-7dcf89b98b-7cjph"
Oct 30 22:21:14.276869 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:14.276848    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8cbd04b4-bb3e-4bbb-a41a-46131c9de051-dshm\") pod \"distinct-2-w0-09d0d5-kserve-7dcf89b98b-7cjph\" (UID: \"8cbd04b4-bb3e-4bbb-a41a-46131c9de051\") " pod="e2e-models-e2e-worker-w0-09d0d5/distinct-2-w0-09d0d5-kserve-7dcf89b98b-7cjph"
Oct 30 22:21:14.277010 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:14.276989    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8cbd04b4-bb3e-4bbb-a41a-46131c9de051-tls-certs\") pod \"distinct-2-w0-09d0d5-kserve-7dcf89b98b-7cjph\" (UID: \"8cbd04b4-bb3e-4bbb-a41a-46131c9de051\") " pod="e2e-models-e2e-worker-w0-09d0d5/distinct-2-w0-09d0d5-kserve-7dcf89b98b-7cjph"
Oct 30 22:21:14.291996 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:14.291970    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5zq8\" (UniqueName: \"kubernetes.io/projected/8cbd04b4-bb3e-4bbb-a41a-46131c9de051-kube-api-access-h5zq8\") pod \"distinct-2-w0-09d0d5-kserve-7dcf89b98b-7cjph\" (UID: \"8cbd04b4-bb3e-4bbb-a41a-46131c9de051\") " pod="e2e-models-e2e-worker-w0-09d0d5/distinct-2-w0-09d0d5-kserve-7dcf89b98b-7cjph"
Oct 30 22:21:14.434321 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:14.434279    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-09d0d5/distinct-2-w0-09d0d5-kserve-7dcf89b98b-7cjph"
Oct 30 22:21:14.752146 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:14.752074    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-09d0d5/distinct-2-w0-09d0d5-kserve-7dcf89b98b-7cjph"]
Oct 30 22:21:14.756290 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:21:14.756264    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8cbd04b4_bb3e_4bbb_a41a_46131c9de051.slice/crio-22c1fed02d98268648e6f0f1a020908fd42d87d3137ae37a17d00f5c0d2b0d54 WatchSource:0}: Error finding container 22c1fed02d98268648e6f0f1a020908fd42d87d3137ae37a17d00f5c0d2b0d54: Status 404 returned error can't find the container with id 22c1fed02d98268648e6f0f1a020908fd42d87d3137ae37a17d00f5c0d2b0d54
Oct 30 22:21:14.982350 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:14.982317    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-09d0d5/distinct-2-w0-09d0d5-kserve-7dcf89b98b-7cjph" event={"ID":"8cbd04b4-bb3e-4bbb-a41a-46131c9de051","Type":"ContainerStarted","Data":"557d03a6b2a3f5cc3860ee8a21e35b522211143c14a9f15f58380cf9e364a499"}
Oct 30 22:21:14.982524 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:14.982360    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-09d0d5/distinct-2-w0-09d0d5-kserve-7dcf89b98b-7cjph" event={"ID":"8cbd04b4-bb3e-4bbb-a41a-46131c9de051","Type":"ContainerStarted","Data":"22c1fed02d98268648e6f0f1a020908fd42d87d3137ae37a17d00f5c0d2b0d54"}
Oct 30 22:21:15.022218 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:15.022111    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-09d0d5/distinct-2-w0-09d0d5-kserve-7dcf89b98b-7cjph" podStartSLOduration=1.022093907 podStartE2EDuration="1.022093907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:21:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:21:15.019952663 +0000 UTC m=+858.945878880" watchObservedRunningTime="2026-10-30 22:21:15.022093907 +0000 UTC m=+858.948020114"
Oct 30 22:21:15.435241 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:15.435212    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-09d0d5/distinct-2-w0-09d0d5-kserve-7dcf89b98b-7cjph"
Oct 30 22:21:22.601274 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:22.601239    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-tbj69"]
Oct 30 22:21:22.609002 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:22.608969    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-tbj69"
Oct 30 22:21:22.627815 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:22.627786    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-tbj69"]
Oct 30 22:21:22.630963 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:22.630936    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-gt2jk"]
Oct 30 22:21:22.635240 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:22.635210    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-gt2jk"
Oct 30 22:21:22.669421 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:22.669392    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-gt2jk"]
Oct 30 22:21:22.778846 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:22.778813    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5d9c5bf-3e60-40b7-9dc0-3588b27074b8-workload-socket\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-tbj69\" (UID: \"f5d9c5bf-3e60-40b7-9dc0-3588b27074b8\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-tbj69"
Oct 30 22:21:22.779016 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:22.778864    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5d9c5bf-3e60-40b7-9dc0-3588b27074b8-credential-socket\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-tbj69\" (UID: \"f5d9c5bf-3e60-40b7-9dc0-3588b27074b8\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-tbj69"
Oct 30 22:21:22.779092 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:22.779007    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5d9c5bf-3e60-40b7-9dc0-3588b27074b8-istio-envoy\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-tbj69\" (UID: \"f5d9c5bf-3e60-40b7-9dc0-3588b27074b8\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-tbj69"
Oct 30 22:21:22.779092 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:22.779069    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5d9c5bf-3e60-40b7-9dc0-3588b27074b8-istio-token\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-tbj69\" (UID: \"f5d9c5bf-3e60-40b7-9dc0-3588b27074b8\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-tbj69"
Oct 30 22:21:22.779165 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:22.779096    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92d165b0-c272-4b3e-969d-ad9dfa6e84d9-credential-socket\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-gt2jk\" (UID: \"92d165b0-c272-4b3e-969d-ad9dfa6e84d9\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-gt2jk"
Oct 30 22:21:22.779198 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:22.779188    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4s27\" (UniqueName: \"kubernetes.io/projected/f5d9c5bf-3e60-40b7-9dc0-3588b27074b8-kube-api-access-w4s27\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-tbj69\" (UID: \"f5d9c5bf-3e60-40b7-9dc0-3588b27074b8\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-tbj69"
Oct 30 22:21:22.779230 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:22.779219    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5d9c5bf-3e60-40b7-9dc0-3588b27074b8-workload-certs\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-tbj69\" (UID: \"f5d9c5bf-3e60-40b7-9dc0-3588b27074b8\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-tbj69"
Oct 30 22:21:22.779270 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:22.779239    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92d165b0-c272-4b3e-969d-ad9dfa6e84d9-workload-socket\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-gt2jk\" (UID: \"92d165b0-c272-4b3e-969d-ad9dfa6e84d9\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-gt2jk"
Oct 30 22:21:22.779310 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:22.779296    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbwff\" (UniqueName: \"kubernetes.io/projected/92d165b0-c272-4b3e-969d-ad9dfa6e84d9-kube-api-access-lbwff\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-gt2jk\" (UID: \"92d165b0-c272-4b3e-969d-ad9dfa6e84d9\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-gt2jk"
Oct 30 22:21:22.779356 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:22.779338    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92d165b0-c272-4b3e-969d-ad9dfa6e84d9-istio-token\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-gt2jk\" (UID: \"92d165b0-c272-4b3e-969d-ad9dfa6e84d9\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-gt2jk"
Oct 30 22:21:22.779441 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:22.779426    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92d165b0-c272-4b3e-969d-ad9dfa6e84d9-workload-certs\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-gt2jk\" (UID: \"92d165b0-c272-4b3e-969d-ad9dfa6e84d9\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-gt2jk"
Oct 30 22:21:22.779495 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:22.779469    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92d165b0-c272-4b3e-969d-ad9dfa6e84d9-istio-envoy\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-gt2jk\" (UID: \"92d165b0-c272-4b3e-969d-ad9dfa6e84d9\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-gt2jk"
Oct 30 22:21:22.779545 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:22.779493    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5d9c5bf-3e60-40b7-9dc0-3588b27074b8-istiod-ca-cert\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-tbj69\" (UID: \"f5d9c5bf-3e60-40b7-9dc0-3588b27074b8\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-tbj69"
Oct 30 22:21:22.779644 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:22.779624    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5d9c5bf-3e60-40b7-9dc0-3588b27074b8-istio-podinfo\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-tbj69\" (UID: \"f5d9c5bf-3e60-40b7-9dc0-3588b27074b8\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-tbj69"
Oct 30 22:21:22.779700 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:22.779687    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92d165b0-c272-4b3e-969d-ad9dfa6e84d9-istio-data\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-gt2jk\" (UID: \"92d165b0-c272-4b3e-969d-ad9dfa6e84d9\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-gt2jk"
Oct 30 22:21:22.779755 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:22.779717    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92d165b0-c272-4b3e-969d-ad9dfa6e84d9-istio-podinfo\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-gt2jk\" (UID: \"92d165b0-c272-4b3e-969d-ad9dfa6e84d9\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-gt2jk"
Oct 30 22:21:22.779805 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:22.779751    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92d165b0-c272-4b3e-969d-ad9dfa6e84d9-istiod-ca-cert\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-gt2jk\" (UID: \"92d165b0-c272-4b3e-969d-ad9dfa6e84d9\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-gt2jk"
Oct 30 22:21:22.779861 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:22.779841    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5d9c5bf-3e60-40b7-9dc0-3588b27074b8-istio-data\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-tbj69\" (UID: \"f5d9c5bf-3e60-40b7-9dc0-3588b27074b8\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-tbj69"
Oct 30 22:21:22.881521 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:22.881489    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5d9c5bf-3e60-40b7-9dc0-3588b27074b8-workload-socket\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-tbj69\" (UID: \"f5d9c5bf-3e60-40b7-9dc0-3588b27074b8\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-tbj69"
Oct 30 22:21:22.881674 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:22.881545    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5d9c5bf-3e60-40b7-9dc0-3588b27074b8-credential-socket\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-tbj69\" (UID: \"f5d9c5bf-3e60-40b7-9dc0-3588b27074b8\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-tbj69"
Oct 30 22:21:22.881674 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:22.881594    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5d9c5bf-3e60-40b7-9dc0-3588b27074b8-istio-envoy\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-tbj69\" (UID: \"f5d9c5bf-3e60-40b7-9dc0-3588b27074b8\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-tbj69"
Oct 30 22:21:22.881674 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:22.881631    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5d9c5bf-3e60-40b7-9dc0-3588b27074b8-istio-token\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-tbj69\" (UID: \"f5d9c5bf-3e60-40b7-9dc0-3588b27074b8\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-tbj69"
Oct 30 22:21:22.881838 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:22.881721    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92d165b0-c272-4b3e-969d-ad9dfa6e84d9-credential-socket\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-gt2jk\" (UID: \"92d165b0-c272-4b3e-969d-ad9dfa6e84d9\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-gt2jk"
Oct 30 22:21:22.881838 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:22.881821    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4s27\" (UniqueName: \"kubernetes.io/projected/f5d9c5bf-3e60-40b7-9dc0-3588b27074b8-kube-api-access-w4s27\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-tbj69\" (UID: \"f5d9c5bf-3e60-40b7-9dc0-3588b27074b8\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-tbj69"
Oct 30 22:21:22.881941 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:22.881872    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5d9c5bf-3e60-40b7-9dc0-3588b27074b8-workload-certs\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-tbj69\" (UID: \"f5d9c5bf-3e60-40b7-9dc0-3588b27074b8\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-tbj69"
Oct 30 22:21:22.881996 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:22.881973    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5d9c5bf-3e60-40b7-9dc0-3588b27074b8-workload-socket\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-tbj69\" (UID: \"f5d9c5bf-3e60-40b7-9dc0-3588b27074b8\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-tbj69"
Oct 30 22:21:22.882057 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:22.881996    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92d165b0-c272-4b3e-969d-ad9dfa6e84d9-workload-socket\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-gt2jk\" (UID: \"92d165b0-c272-4b3e-969d-ad9dfa6e84d9\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-gt2jk"
Oct 30 22:21:22.882118 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:22.882057    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbwff\" (UniqueName: \"kubernetes.io/projected/92d165b0-c272-4b3e-969d-ad9dfa6e84d9-kube-api-access-lbwff\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-gt2jk\" (UID: \"92d165b0-c272-4b3e-969d-ad9dfa6e84d9\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-gt2jk"
Oct 30 22:21:22.882118 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:22.882108    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92d165b0-c272-4b3e-969d-ad9dfa6e84d9-istio-token\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-gt2jk\" (UID: \"92d165b0-c272-4b3e-969d-ad9dfa6e84d9\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-gt2jk"
Oct 30 22:21:22.882234 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:22.882160    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5d9c5bf-3e60-40b7-9dc0-3588b27074b8-credential-socket\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-tbj69\" (UID: \"f5d9c5bf-3e60-40b7-9dc0-3588b27074b8\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-tbj69"
Oct 30 22:21:22.882234 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:22.882170    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92d165b0-c272-4b3e-969d-ad9dfa6e84d9-workload-certs\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-gt2jk\" (UID: \"92d165b0-c272-4b3e-969d-ad9dfa6e84d9\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-gt2jk"
Oct 30 22:21:22.882234 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:22.882226    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92d165b0-c272-4b3e-969d-ad9dfa6e84d9-istio-envoy\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-gt2jk\" (UID: \"92d165b0-c272-4b3e-969d-ad9dfa6e84d9\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-gt2jk"
Oct 30 22:21:22.882396 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:22.882231    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92d165b0-c272-4b3e-969d-ad9dfa6e84d9-credential-socket\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-gt2jk\" (UID: \"92d165b0-c272-4b3e-969d-ad9dfa6e84d9\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-gt2jk"
Oct 30 22:21:22.882396 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:22.882257    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5d9c5bf-3e60-40b7-9dc0-3588b27074b8-istiod-ca-cert\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-tbj69\" (UID: \"f5d9c5bf-3e60-40b7-9dc0-3588b27074b8\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-tbj69"
Oct 30 22:21:22.882396 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:22.882301    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5d9c5bf-3e60-40b7-9dc0-3588b27074b8-istio-podinfo\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-tbj69\" (UID: \"f5d9c5bf-3e60-40b7-9dc0-3588b27074b8\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-tbj69"
Oct 30 22:21:22.882396 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:22.882377    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92d165b0-c272-4b3e-969d-ad9dfa6e84d9-istio-data\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-gt2jk\" (UID: \"92d165b0-c272-4b3e-969d-ad9dfa6e84d9\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-gt2jk"
Oct 30 22:21:22.882614 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:22.882396    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5d9c5bf-3e60-40b7-9dc0-3588b27074b8-workload-certs\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-tbj69\" (UID: \"f5d9c5bf-3e60-40b7-9dc0-3588b27074b8\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-tbj69"
Oct 30 22:21:22.882614 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:22.882395    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92d165b0-c272-4b3e-969d-ad9dfa6e84d9-workload-socket\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-gt2jk\" (UID: \"92d165b0-c272-4b3e-969d-ad9dfa6e84d9\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-gt2jk"
Oct 30 22:21:22.882614 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:22.882401    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92d165b0-c272-4b3e-969d-ad9dfa6e84d9-istio-podinfo\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-gt2jk\" (UID: \"92d165b0-c272-4b3e-969d-ad9dfa6e84d9\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-gt2jk"
Oct 30 22:21:22.882614 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:22.882526    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92d165b0-c272-4b3e-969d-ad9dfa6e84d9-istiod-ca-cert\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-gt2jk\" (UID: \"92d165b0-c272-4b3e-969d-ad9dfa6e84d9\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-gt2jk"
Oct 30 22:21:22.882614 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:22.882477    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92d165b0-c272-4b3e-969d-ad9dfa6e84d9-workload-certs\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-gt2jk\" (UID: \"92d165b0-c272-4b3e-969d-ad9dfa6e84d9\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-gt2jk"
Oct 30 22:21:22.882614 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:22.882559    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5d9c5bf-3e60-40b7-9dc0-3588b27074b8-istio-data\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-tbj69\" (UID: \"f5d9c5bf-3e60-40b7-9dc0-3588b27074b8\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-tbj69"
Oct 30 22:21:22.882907 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:22.882821    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5d9c5bf-3e60-40b7-9dc0-3588b27074b8-istio-data\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-tbj69\" (UID: \"f5d9c5bf-3e60-40b7-9dc0-3588b27074b8\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-tbj69"
Oct 30 22:21:22.883104 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:22.883079    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5d9c5bf-3e60-40b7-9dc0-3588b27074b8-istiod-ca-cert\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-tbj69\" (UID: \"f5d9c5bf-3e60-40b7-9dc0-3588b27074b8\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-tbj69"
Oct 30 22:21:22.883324 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:22.883174    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92d165b0-c272-4b3e-969d-ad9dfa6e84d9-istio-data\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-gt2jk\" (UID: \"92d165b0-c272-4b3e-969d-ad9dfa6e84d9\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-gt2jk"
Oct 30 22:21:22.883403 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:22.883389    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92d165b0-c272-4b3e-969d-ad9dfa6e84d9-istiod-ca-cert\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-gt2jk\" (UID: \"92d165b0-c272-4b3e-969d-ad9dfa6e84d9\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-gt2jk"
Oct 30 22:21:22.884605 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:22.884585    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5d9c5bf-3e60-40b7-9dc0-3588b27074b8-istio-envoy\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-tbj69\" (UID: \"f5d9c5bf-3e60-40b7-9dc0-3588b27074b8\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-tbj69"
Oct 30 22:21:22.884996 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:22.884976    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92d165b0-c272-4b3e-969d-ad9dfa6e84d9-istio-podinfo\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-gt2jk\" (UID: \"92d165b0-c272-4b3e-969d-ad9dfa6e84d9\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-gt2jk"
Oct 30 22:21:22.885119 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:22.884980    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5d9c5bf-3e60-40b7-9dc0-3588b27074b8-istio-podinfo\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-tbj69\" (UID: \"f5d9c5bf-3e60-40b7-9dc0-3588b27074b8\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-tbj69"
Oct 30 22:21:22.885119 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:22.885016    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92d165b0-c272-4b3e-969d-ad9dfa6e84d9-istio-envoy\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-gt2jk\" (UID: \"92d165b0-c272-4b3e-969d-ad9dfa6e84d9\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-gt2jk"
Oct 30 22:21:22.899876 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:22.899852    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbwff\" (UniqueName: \"kubernetes.io/projected/92d165b0-c272-4b3e-969d-ad9dfa6e84d9-kube-api-access-lbwff\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-gt2jk\" (UID: \"92d165b0-c272-4b3e-969d-ad9dfa6e84d9\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-gt2jk"
Oct 30 22:21:22.900114 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:22.900088    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92d165b0-c272-4b3e-969d-ad9dfa6e84d9-istio-token\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-gt2jk\" (UID: \"92d165b0-c272-4b3e-969d-ad9dfa6e84d9\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-gt2jk"
Oct 30 22:21:22.900410 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:22.900392    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5d9c5bf-3e60-40b7-9dc0-3588b27074b8-istio-token\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-tbj69\" (UID: \"f5d9c5bf-3e60-40b7-9dc0-3588b27074b8\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-tbj69"
Oct 30 22:21:22.900761 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:22.900741    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4s27\" (UniqueName: \"kubernetes.io/projected/f5d9c5bf-3e60-40b7-9dc0-3588b27074b8-kube-api-access-w4s27\") pod \"e2e-worker-w3-d52c59-openshift-default-5466968fb7-tbj69\" (UID: \"f5d9c5bf-3e60-40b7-9dc0-3588b27074b8\") " pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-tbj69"
Oct 30 22:21:22.924418 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:22.924397    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-tbj69"
Oct 30 22:21:22.952684 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:22.952663    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-gt2jk"
Oct 30 22:21:23.241008 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:23.240925    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-tbj69"]
Oct 30 22:21:23.245937 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:21:23.245911    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5d9c5bf_3e60_40b7_9dc0_3588b27074b8.slice/crio-379406bf0be7800f8cd5edceb52a4be4603bd1b7883ea8554ecf120bc869a342 WatchSource:0}: Error finding container 379406bf0be7800f8cd5edceb52a4be4603bd1b7883ea8554ecf120bc869a342: Status 404 returned error can't find the container with id 379406bf0be7800f8cd5edceb52a4be4603bd1b7883ea8554ecf120bc869a342
Oct 30 22:21:23.248536 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:23.248491    4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:21:23.248623 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:23.248569    4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:21:23.248623 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:23.248603    4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:21:23.280764 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:23.280740    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-gt2jk"]
Oct 30 22:21:23.284014 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:21:23.283991    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92d165b0_c272_4b3e_969d_ad9dfa6e84d9.slice/crio-1afb1e0c09e8a23f2a4401882c31ba0b696eec4befce28db4984bf9a0c9d6330 WatchSource:0}: Error finding container 1afb1e0c09e8a23f2a4401882c31ba0b696eec4befce28db4984bf9a0c9d6330: Status 404 returned error can't find the container with id 1afb1e0c09e8a23f2a4401882c31ba0b696eec4befce28db4984bf9a0c9d6330
Oct 30 22:21:23.286929 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:23.286895    4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:21:23.287000 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:23.286985    4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:21:23.287059 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:23.287029    4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:21:24.036694 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:24.036655    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-gt2jk" event={"ID":"92d165b0-c272-4b3e-969d-ad9dfa6e84d9","Type":"ContainerStarted","Data":"93d0b2f93136711a14133fa831994d108dd7619ee09468d75a601b44c0a3241d"}
Oct 30 22:21:24.037128 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:24.036720    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-gt2jk" event={"ID":"92d165b0-c272-4b3e-969d-ad9dfa6e84d9","Type":"ContainerStarted","Data":"1afb1e0c09e8a23f2a4401882c31ba0b696eec4befce28db4984bf9a0c9d6330"}
Oct 30 22:21:24.038374 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:24.038349    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-tbj69" event={"ID":"f5d9c5bf-3e60-40b7-9dc0-3588b27074b8","Type":"ContainerStarted","Data":"b302ec196a15dbe48c2859dbad3a0f514c1fbb086e51a0467efcba706bc26534"}
Oct 30 22:21:24.038504 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:24.038388    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-tbj69" event={"ID":"f5d9c5bf-3e60-40b7-9dc0-3588b27074b8","Type":"ContainerStarted","Data":"379406bf0be7800f8cd5edceb52a4be4603bd1b7883ea8554ecf120bc869a342"}
Oct 30 22:21:24.123388 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:24.123331    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-gt2jk" podStartSLOduration=2.123314715 podStartE2EDuration="2.123314715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:21:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:21:24.120822586 +0000 UTC m=+868.046748795" watchObservedRunningTime="2026-10-30 22:21:24.123314715 +0000 UTC m=+868.049241000"
Oct 30 22:21:24.168759 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:24.167056    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-tbj69" podStartSLOduration=2.167021916 podStartE2EDuration="2.167021916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:21:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:21:24.165295997 +0000 UTC m=+868.091222216" watchObservedRunningTime="2026-10-30 22:21:24.167021916 +0000 UTC m=+868.092948122"
Oct 30 22:21:24.434878 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:24.434839    4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-09d0d5/distinct-2-w0-09d0d5-kserve-7dcf89b98b-7cjph"
Oct 30 22:21:24.447797 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:24.447771    4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-09d0d5/distinct-2-w0-09d0d5-kserve-7dcf89b98b-7cjph"
Oct 30 22:21:24.929061 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:24.928388    4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-tbj69"
Oct 30 22:21:24.929962 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:24.929684    4477 patch_prober.go:29] interesting pod/e2e-worker-w3-d52c59-openshift-default-5466968fb7-tbj69 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Oct 30 22:21:24.929962 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:24.929733    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-tbj69" podUID="f5d9c5bf-3e60-40b7-9dc0-3588b27074b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Oct 30 22:21:24.953470 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:24.953450    4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-gt2jk"
Oct 30 22:21:24.954475 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:24.954456    4477 patch_prober.go:29] interesting pod/e2e-worker-w3-d52c59-openshift-default-5466968fb7-gt2jk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Oct 30 22:21:24.954561 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:24.954510    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-gt2jk" podUID="92d165b0-c272-4b3e-969d-ad9dfa6e84d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Oct 30 22:21:25.056320 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:25.056293    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-09d0d5/distinct-2-w0-09d0d5-kserve-7dcf89b98b-7cjph"
Oct 30 22:21:25.928065 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:25.927672    4477 patch_prober.go:29] interesting pod/e2e-worker-w3-d52c59-openshift-default-5466968fb7-tbj69 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Oct 30 22:21:25.928065 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:25.927733    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-tbj69" podUID="f5d9c5bf-3e60-40b7-9dc0-3588b27074b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Oct 30 22:21:25.953461 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:25.953429    4477 patch_prober.go:29] interesting pod/e2e-worker-w3-d52c59-openshift-default-5466968fb7-gt2jk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Oct 30 22:21:25.953618 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:25.953474    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-gt2jk" podUID="92d165b0-c272-4b3e-969d-ad9dfa6e84d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Oct 30 22:21:26.266715 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:21:26.266594    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 22:21:26.925702 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:26.925648    4477 patch_prober.go:29] interesting pod/e2e-worker-w3-d52c59-openshift-default-5466968fb7-tbj69 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Oct 30 22:21:26.925907 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:26.925717    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-tbj69" podUID="f5d9c5bf-3e60-40b7-9dc0-3588b27074b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Oct 30 22:21:26.953054 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:26.953016    4477 patch_prober.go:29] interesting pod/e2e-worker-w3-d52c59-openshift-default-5466968fb7-gt2jk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Oct 30 22:21:26.953195 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:26.953082    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-gt2jk" podUID="92d165b0-c272-4b3e-969d-ad9dfa6e84d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Oct 30 22:21:27.928258 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:27.928062    4477 patch_prober.go:29] interesting pod/e2e-worker-w3-d52c59-openshift-default-5466968fb7-tbj69 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Oct 30 22:21:27.928258 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:27.928116    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-tbj69" podUID="f5d9c5bf-3e60-40b7-9dc0-3588b27074b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Oct 30 22:21:27.953702 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:27.953676    4477 patch_prober.go:29] interesting pod/e2e-worker-w3-d52c59-openshift-default-5466968fb7-gt2jk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Oct 30 22:21:27.953845 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:27.953728    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-gt2jk" podUID="92d165b0-c272-4b3e-969d-ad9dfa6e84d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Oct 30 22:21:28.928061 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:28.927360    4477 patch_prober.go:29] interesting pod/e2e-worker-w3-d52c59-openshift-default-5466968fb7-tbj69 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Oct 30 22:21:28.928061 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:28.927424    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-tbj69" podUID="f5d9c5bf-3e60-40b7-9dc0-3588b27074b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Oct 30 22:21:28.953171 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:28.953141    4477 patch_prober.go:29] interesting pod/e2e-worker-w3-d52c59-openshift-default-5466968fb7-gt2jk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Oct 30 22:21:28.953528 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:28.953183    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-gt2jk" podUID="92d165b0-c272-4b3e-969d-ad9dfa6e84d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Oct 30 22:21:29.925742 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:29.925699    4477 patch_prober.go:29] interesting pod/e2e-worker-w3-d52c59-openshift-default-5466968fb7-tbj69 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Oct 30 22:21:29.925912 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:29.925772    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-tbj69" podUID="f5d9c5bf-3e60-40b7-9dc0-3588b27074b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Oct 30 22:21:29.953521 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:29.953490    4477 patch_prober.go:29] interesting pod/e2e-worker-w3-d52c59-openshift-default-5466968fb7-gt2jk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Oct 30 22:21:29.953877 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:29.953543    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-gt2jk" podUID="92d165b0-c272-4b3e-969d-ad9dfa6e84d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Oct 30 22:21:30.967973 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:30.967946    4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-tbj69"
Oct 30 22:21:30.969433 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:30.969293    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-tbj69"
Oct 30 22:21:30.974380 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:30.974355    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-tbj69"
Oct 30 22:21:31.333760 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:31.333655    4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-gt2jk"
Oct 30 22:21:31.334112 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:31.334088    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-gt2jk"
Oct 30 22:21:31.335441 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:31.335416    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-gt2jk"
Oct 30 22:21:37.916970 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:37.916940    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-9j5tr"]
Oct 30 22:21:37.921469 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:37.921446    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-9j5tr"
Oct 30 22:21:37.950143 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:37.950116    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-9j5tr"]
Oct 30 22:21:38.062600 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:38.062560    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-sqxvr"]
Oct 30 22:21:38.076861 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:38.076827    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fb811da-4dc2-48bb-bec6-afb7a1a80ed2-istio-data\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-9j5tr\" (UID: \"1fb811da-4dc2-48bb-bec6-afb7a1a80ed2\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-9j5tr"
Oct 30 22:21:38.077054 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:38.076882    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fb811da-4dc2-48bb-bec6-afb7a1a80ed2-istio-podinfo\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-9j5tr\" (UID: \"1fb811da-4dc2-48bb-bec6-afb7a1a80ed2\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-9j5tr"
Oct 30 22:21:38.077054 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:38.077018    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fb811da-4dc2-48bb-bec6-afb7a1a80ed2-istio-token\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-9j5tr\" (UID: \"1fb811da-4dc2-48bb-bec6-afb7a1a80ed2\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-9j5tr"
Oct 30 22:21:38.077186 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:38.077091    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fb811da-4dc2-48bb-bec6-afb7a1a80ed2-istio-envoy\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-9j5tr\" (UID: \"1fb811da-4dc2-48bb-bec6-afb7a1a80ed2\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-9j5tr"
Oct 30 22:21:38.077186 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:38.077151    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fb811da-4dc2-48bb-bec6-afb7a1a80ed2-workload-socket\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-9j5tr\" (UID: \"1fb811da-4dc2-48bb-bec6-afb7a1a80ed2\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-9j5tr"
Oct 30 22:21:38.077259 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:38.077232    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqqcm\" (UniqueName: \"kubernetes.io/projected/1fb811da-4dc2-48bb-bec6-afb7a1a80ed2-kube-api-access-rqqcm\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-9j5tr\" (UID: \"1fb811da-4dc2-48bb-bec6-afb7a1a80ed2\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-9j5tr"
Oct 30 22:21:38.077333 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:38.077317    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fb811da-4dc2-48bb-bec6-afb7a1a80ed2-workload-certs\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-9j5tr\" (UID: \"1fb811da-4dc2-48bb-bec6-afb7a1a80ed2\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-9j5tr"
Oct 30 22:21:38.077445 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:38.077424    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fb811da-4dc2-48bb-bec6-afb7a1a80ed2-credential-socket\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-9j5tr\" (UID: \"1fb811da-4dc2-48bb-bec6-afb7a1a80ed2\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-9j5tr"
Oct 30 22:21:38.077485 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:38.077454    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fb811da-4dc2-48bb-bec6-afb7a1a80ed2-istiod-ca-cert\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-9j5tr\" (UID: \"1fb811da-4dc2-48bb-bec6-afb7a1a80ed2\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-9j5tr"
Oct 30 22:21:38.079270 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:38.079251    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-sqxvr"
Oct 30 22:21:38.088167 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:38.088140    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-sqxvr"]
Oct 30 22:21:38.091413 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:38.091390    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-m74tl"]
Oct 30 22:21:38.095373 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:38.095354    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-m74tl"
Oct 30 22:21:38.117478 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:38.116902    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-m74tl"]
Oct 30 22:21:38.178379 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:38.178310    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fb811da-4dc2-48bb-bec6-afb7a1a80ed2-istio-podinfo\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-9j5tr\" (UID: \"1fb811da-4dc2-48bb-bec6-afb7a1a80ed2\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-9j5tr"
Oct 30 22:21:38.178509 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:38.178382    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fb811da-4dc2-48bb-bec6-afb7a1a80ed2-istio-token\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-9j5tr\" (UID: \"1fb811da-4dc2-48bb-bec6-afb7a1a80ed2\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-9j5tr"
Oct 30 22:21:38.178509 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:38.178413    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fb811da-4dc2-48bb-bec6-afb7a1a80ed2-istio-envoy\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-9j5tr\" (UID: \"1fb811da-4dc2-48bb-bec6-afb7a1a80ed2\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-9j5tr"
Oct 30 22:21:38.178509 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:38.178445    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/909907e9-319a-48dc-90d6-735cee614217-istiod-ca-cert\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-sqxvr\" (UID: \"909907e9-319a-48dc-90d6-735cee614217\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-sqxvr"
Oct 30 22:21:38.178509 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:38.178477    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/909907e9-319a-48dc-90d6-735cee614217-istio-envoy\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-sqxvr\" (UID: \"909907e9-319a-48dc-90d6-735cee614217\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-sqxvr"
Oct 30 22:21:38.178720 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:38.178521    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fb811da-4dc2-48bb-bec6-afb7a1a80ed2-workload-socket\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-9j5tr\" (UID: \"1fb811da-4dc2-48bb-bec6-afb7a1a80ed2\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-9j5tr"
Oct 30 22:21:38.178720 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:38.178552    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqqcm\" (UniqueName: \"kubernetes.io/projected/1fb811da-4dc2-48bb-bec6-afb7a1a80ed2-kube-api-access-rqqcm\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-9j5tr\" (UID: \"1fb811da-4dc2-48bb-bec6-afb7a1a80ed2\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-9j5tr"
Oct 30 22:21:38.178720 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:38.178632    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/909907e9-319a-48dc-90d6-735cee614217-workload-certs\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-sqxvr\" (UID: \"909907e9-319a-48dc-90d6-735cee614217\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-sqxvr"
Oct 30 22:21:38.178720 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:38.178707    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/909907e9-319a-48dc-90d6-735cee614217-istio-data\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-sqxvr\" (UID: \"909907e9-319a-48dc-90d6-735cee614217\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-sqxvr"
Oct 30 22:21:38.178913 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:38.178751    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/909907e9-319a-48dc-90d6-735cee614217-workload-socket\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-sqxvr\" (UID: \"909907e9-319a-48dc-90d6-735cee614217\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-sqxvr"
Oct 30 22:21:38.178913 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:38.178791    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fb811da-4dc2-48bb-bec6-afb7a1a80ed2-workload-certs\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-9j5tr\" (UID: \"1fb811da-4dc2-48bb-bec6-afb7a1a80ed2\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-9j5tr"
Oct 30 22:21:38.178913 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:38.178886    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/909907e9-319a-48dc-90d6-735cee614217-istio-token\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-sqxvr\" (UID: \"909907e9-319a-48dc-90d6-735cee614217\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-sqxvr"
Oct 30 22:21:38.179233 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:38.179212    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fb811da-4dc2-48bb-bec6-afb7a1a80ed2-workload-certs\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-9j5tr\" (UID: \"1fb811da-4dc2-48bb-bec6-afb7a1a80ed2\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-9j5tr"
Oct 30 22:21:38.179308 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:38.179289    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fb811da-4dc2-48bb-bec6-afb7a1a80ed2-workload-socket\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-9j5tr\" (UID: \"1fb811da-4dc2-48bb-bec6-afb7a1a80ed2\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-9j5tr"
Oct 30 22:21:38.179354 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:38.179336    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/909907e9-319a-48dc-90d6-735cee614217-istio-podinfo\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-sqxvr\" (UID: \"909907e9-319a-48dc-90d6-735cee614217\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-sqxvr"
Oct 30 22:21:38.179416 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:38.179399    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fb811da-4dc2-48bb-bec6-afb7a1a80ed2-credential-socket\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-9j5tr\" (UID: \"1fb811da-4dc2-48bb-bec6-afb7a1a80ed2\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-9j5tr"
Oct 30 22:21:38.179460 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:38.179441    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fb811da-4dc2-48bb-bec6-afb7a1a80ed2-istiod-ca-cert\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-9j5tr\" (UID: \"1fb811da-4dc2-48bb-bec6-afb7a1a80ed2\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-9j5tr"
Oct 30 22:21:38.179738 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:38.179719    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fb811da-4dc2-48bb-bec6-afb7a1a80ed2-credential-socket\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-9j5tr\" (UID: \"1fb811da-4dc2-48bb-bec6-afb7a1a80ed2\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-9j5tr"
Oct 30 22:21:38.179836 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:38.179783    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/909907e9-319a-48dc-90d6-735cee614217-credential-socket\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-sqxvr\" (UID: \"909907e9-319a-48dc-90d6-735cee614217\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-sqxvr"
Oct 30 22:21:38.179897 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:38.179842    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvrt2\" (UniqueName: \"kubernetes.io/projected/909907e9-319a-48dc-90d6-735cee614217-kube-api-access-vvrt2\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-sqxvr\" (UID: \"909907e9-319a-48dc-90d6-735cee614217\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-sqxvr"
Oct 30 22:21:38.179897 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:38.179880    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fb811da-4dc2-48bb-bec6-afb7a1a80ed2-istio-data\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-9j5tr\" (UID: \"1fb811da-4dc2-48bb-bec6-afb7a1a80ed2\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-9j5tr"
Oct 30 22:21:38.180158 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:38.180134    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fb811da-4dc2-48bb-bec6-afb7a1a80ed2-istiod-ca-cert\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-9j5tr\" (UID: \"1fb811da-4dc2-48bb-bec6-afb7a1a80ed2\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-9j5tr"
Oct 30 22:21:38.180235 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:38.180165    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fb811da-4dc2-48bb-bec6-afb7a1a80ed2-istio-data\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-9j5tr\" (UID: \"1fb811da-4dc2-48bb-bec6-afb7a1a80ed2\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-9j5tr"
Oct 30 22:21:38.181449 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:38.181428    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fb811da-4dc2-48bb-bec6-afb7a1a80ed2-istio-envoy\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-9j5tr\" (UID: \"1fb811da-4dc2-48bb-bec6-afb7a1a80ed2\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-9j5tr"
Oct 30 22:21:38.181563 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:38.181472    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fb811da-4dc2-48bb-bec6-afb7a1a80ed2-istio-podinfo\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-9j5tr\" (UID: \"1fb811da-4dc2-48bb-bec6-afb7a1a80ed2\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-9j5tr"
Oct 30 22:21:38.197182 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:38.197158    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fb811da-4dc2-48bb-bec6-afb7a1a80ed2-istio-token\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-9j5tr\" (UID: \"1fb811da-4dc2-48bb-bec6-afb7a1a80ed2\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-9j5tr"
Oct 30 22:21:38.197294 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:38.197166    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqqcm\" (UniqueName: \"kubernetes.io/projected/1fb811da-4dc2-48bb-bec6-afb7a1a80ed2-kube-api-access-rqqcm\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-9j5tr\" (UID: \"1fb811da-4dc2-48bb-bec6-afb7a1a80ed2\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-9j5tr"
Oct 30 22:21:38.237809 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:38.237783    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-9j5tr"
Oct 30 22:21:38.281341 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:38.281314    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/909907e9-319a-48dc-90d6-735cee614217-istio-token\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-sqxvr\" (UID: \"909907e9-319a-48dc-90d6-735cee614217\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-sqxvr"
Oct 30 22:21:38.281476 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:38.281354    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e6f6850-1ca2-4f4c-a047-23e35ce2b697-workload-socket\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-m74tl\" (UID: \"3e6f6850-1ca2-4f4c-a047-23e35ce2b697\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-m74tl"
Oct 30 22:21:38.281476 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:38.281382    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/909907e9-319a-48dc-90d6-735cee614217-istio-podinfo\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-sqxvr\" (UID: \"909907e9-319a-48dc-90d6-735cee614217\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-sqxvr"
Oct 30 22:21:38.281476 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:38.281406    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e6f6850-1ca2-4f4c-a047-23e35ce2b697-istiod-ca-cert\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-m74tl\" (UID: \"3e6f6850-1ca2-4f4c-a047-23e35ce2b697\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-m74tl"
Oct 30 22:21:38.281476 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:38.281455    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e6f6850-1ca2-4f4c-a047-23e35ce2b697-istio-token\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-m74tl\" (UID: \"3e6f6850-1ca2-4f4c-a047-23e35ce2b697\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-m74tl"
Oct 30 22:21:38.281685 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:38.281554    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/909907e9-319a-48dc-90d6-735cee614217-credential-socket\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-sqxvr\" (UID: \"909907e9-319a-48dc-90d6-735cee614217\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-sqxvr"
Oct 30 22:21:38.281747 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:38.281686    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvrt2\" (UniqueName: \"kubernetes.io/projected/909907e9-319a-48dc-90d6-735cee614217-kube-api-access-vvrt2\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-sqxvr\" (UID: \"909907e9-319a-48dc-90d6-735cee614217\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-sqxvr"
Oct 30 22:21:38.281805 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:38.281748    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e6f6850-1ca2-4f4c-a047-23e35ce2b697-istio-data\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-m74tl\" (UID: \"3e6f6850-1ca2-4f4c-a047-23e35ce2b697\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-m74tl"
Oct 30 22:21:38.281899 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:38.281870    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e6f6850-1ca2-4f4c-a047-23e35ce2b697-workload-certs\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-m74tl\" (UID: \"3e6f6850-1ca2-4f4c-a047-23e35ce2b697\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-m74tl"
Oct 30 22:21:38.281980 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:38.281925    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e6f6850-1ca2-4f4c-a047-23e35ce2b697-istio-envoy\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-m74tl\" (UID: \"3e6f6850-1ca2-4f4c-a047-23e35ce2b697\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-m74tl"
Oct 30 22:21:38.281980 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:38.281961    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/909907e9-319a-48dc-90d6-735cee614217-credential-socket\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-sqxvr\" (UID: \"909907e9-319a-48dc-90d6-735cee614217\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-sqxvr"
Oct 30 22:21:38.282099 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:38.281984    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/909907e9-319a-48dc-90d6-735cee614217-istiod-ca-cert\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-sqxvr\" (UID: \"909907e9-319a-48dc-90d6-735cee614217\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-sqxvr"
Oct 30 22:21:38.282099 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:38.282017    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/909907e9-319a-48dc-90d6-735cee614217-istio-envoy\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-sqxvr\" (UID: \"909907e9-319a-48dc-90d6-735cee614217\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-sqxvr"
Oct 30 22:21:38.282186 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:38.282093    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e6f6850-1ca2-4f4c-a047-23e35ce2b697-credential-socket\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-m74tl\" (UID: \"3e6f6850-1ca2-4f4c-a047-23e35ce2b697\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-m74tl"
Oct 30 22:21:38.282186 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:38.282122    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/909907e9-319a-48dc-90d6-735cee614217-workload-certs\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-sqxvr\" (UID: \"909907e9-319a-48dc-90d6-735cee614217\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-sqxvr"
Oct 30 22:21:38.282300 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:38.282282    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/909907e9-319a-48dc-90d6-735cee614217-istio-data\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-sqxvr\" (UID: \"909907e9-319a-48dc-90d6-735cee614217\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-sqxvr"
Oct 30 22:21:38.282358 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:38.282318    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nfhf\" (UniqueName: \"kubernetes.io/projected/3e6f6850-1ca2-4f4c-a047-23e35ce2b697-kube-api-access-6nfhf\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-m74tl\" (UID: \"3e6f6850-1ca2-4f4c-a047-23e35ce2b697\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-m74tl"
Oct 30 22:21:38.282417 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:38.282362    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/909907e9-319a-48dc-90d6-735cee614217-workload-socket\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-sqxvr\" (UID: \"909907e9-319a-48dc-90d6-735cee614217\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-sqxvr"
Oct 30 22:21:38.282417 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:38.282397    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e6f6850-1ca2-4f4c-a047-23e35ce2b697-istio-podinfo\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-m74tl\" (UID: \"3e6f6850-1ca2-4f4c-a047-23e35ce2b697\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-m74tl"
Oct 30 22:21:38.282509 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:38.282440    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/909907e9-319a-48dc-90d6-735cee614217-workload-certs\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-sqxvr\" (UID: \"909907e9-319a-48dc-90d6-735cee614217\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-sqxvr"
Oct 30 22:21:38.282639 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:38.282623    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/909907e9-319a-48dc-90d6-735cee614217-istiod-ca-cert\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-sqxvr\" (UID: \"909907e9-319a-48dc-90d6-735cee614217\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-sqxvr"
Oct 30 22:21:38.282679 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:38.282633    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/909907e9-319a-48dc-90d6-735cee614217-istio-data\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-sqxvr\" (UID: \"909907e9-319a-48dc-90d6-735cee614217\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-sqxvr"
Oct 30 22:21:38.282731 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:38.282711    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/909907e9-319a-48dc-90d6-735cee614217-workload-socket\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-sqxvr\" (UID: \"909907e9-319a-48dc-90d6-735cee614217\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-sqxvr"
Oct 30 22:21:38.283866 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:38.283844    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/909907e9-319a-48dc-90d6-735cee614217-istio-podinfo\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-sqxvr\" (UID: \"909907e9-319a-48dc-90d6-735cee614217\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-sqxvr"
Oct 30 22:21:38.284307 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:38.284288    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/909907e9-319a-48dc-90d6-735cee614217-istio-envoy\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-sqxvr\" (UID: \"909907e9-319a-48dc-90d6-735cee614217\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-sqxvr"
Oct 30 22:21:38.303936 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:38.303908    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/909907e9-319a-48dc-90d6-735cee614217-istio-token\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-sqxvr\" (UID: \"909907e9-319a-48dc-90d6-735cee614217\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-sqxvr"
Oct 30 22:21:38.304929 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:38.304908    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvrt2\" (UniqueName: \"kubernetes.io/projected/909907e9-319a-48dc-90d6-735cee614217-kube-api-access-vvrt2\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-sqxvr\" (UID: \"909907e9-319a-48dc-90d6-735cee614217\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-sqxvr"
Oct 30 22:21:38.383345 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:38.383318    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e6f6850-1ca2-4f4c-a047-23e35ce2b697-istio-data\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-m74tl\" (UID: \"3e6f6850-1ca2-4f4c-a047-23e35ce2b697\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-m74tl"
Oct 30 22:21:38.383541 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:38.383381    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e6f6850-1ca2-4f4c-a047-23e35ce2b697-workload-certs\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-m74tl\" (UID: \"3e6f6850-1ca2-4f4c-a047-23e35ce2b697\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-m74tl"
Oct 30 22:21:38.383541 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:38.383404    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e6f6850-1ca2-4f4c-a047-23e35ce2b697-istio-envoy\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-m74tl\" (UID: \"3e6f6850-1ca2-4f4c-a047-23e35ce2b697\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-m74tl"
Oct 30 22:21:38.383541 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:38.383492    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e6f6850-1ca2-4f4c-a047-23e35ce2b697-credential-socket\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-m74tl\" (UID: \"3e6f6850-1ca2-4f4c-a047-23e35ce2b697\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-m74tl"
Oct 30 22:21:38.383716 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:38.383570    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nfhf\" (UniqueName: \"kubernetes.io/projected/3e6f6850-1ca2-4f4c-a047-23e35ce2b697-kube-api-access-6nfhf\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-m74tl\" (UID: \"3e6f6850-1ca2-4f4c-a047-23e35ce2b697\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-m74tl"
Oct 30 22:21:38.383716 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:38.383614    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e6f6850-1ca2-4f4c-a047-23e35ce2b697-istio-podinfo\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-m74tl\" (UID: \"3e6f6850-1ca2-4f4c-a047-23e35ce2b697\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-m74tl"
Oct 30 22:21:38.383831 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:38.383721    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e6f6850-1ca2-4f4c-a047-23e35ce2b697-workload-socket\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-m74tl\" (UID: \"3e6f6850-1ca2-4f4c-a047-23e35ce2b697\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-m74tl"
Oct 30 22:21:38.383831 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:38.383752    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e6f6850-1ca2-4f4c-a047-23e35ce2b697-istio-data\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-m74tl\" (UID: \"3e6f6850-1ca2-4f4c-a047-23e35ce2b697\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-m74tl"
Oct 30 22:21:38.383831 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:38.383765    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e6f6850-1ca2-4f4c-a047-23e35ce2b697-istiod-ca-cert\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-m74tl\" (UID: \"3e6f6850-1ca2-4f4c-a047-23e35ce2b697\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-m74tl"
Oct 30 22:21:38.383831 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:38.383820    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e6f6850-1ca2-4f4c-a047-23e35ce2b697-workload-certs\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-m74tl\" (UID: \"3e6f6850-1ca2-4f4c-a047-23e35ce2b697\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-m74tl"
Oct 30 22:21:38.384020 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:38.383832    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e6f6850-1ca2-4f4c-a047-23e35ce2b697-istio-token\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-m74tl\" (UID: \"3e6f6850-1ca2-4f4c-a047-23e35ce2b697\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-m74tl"
Oct 30 22:21:38.384020 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:38.383928    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e6f6850-1ca2-4f4c-a047-23e35ce2b697-credential-socket\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-m74tl\" (UID: \"3e6f6850-1ca2-4f4c-a047-23e35ce2b697\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-m74tl"
Oct 30 22:21:38.384163 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:38.384056    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e6f6850-1ca2-4f4c-a047-23e35ce2b697-workload-socket\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-m74tl\" (UID: \"3e6f6850-1ca2-4f4c-a047-23e35ce2b697\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-m74tl"
Oct 30 22:21:38.384535 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:38.384508    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e6f6850-1ca2-4f4c-a047-23e35ce2b697-istiod-ca-cert\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-m74tl\" (UID: \"3e6f6850-1ca2-4f4c-a047-23e35ce2b697\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-m74tl"
Oct 30 22:21:38.385672 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:38.385652    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e6f6850-1ca2-4f4c-a047-23e35ce2b697-istio-envoy\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-m74tl\" (UID: \"3e6f6850-1ca2-4f4c-a047-23e35ce2b697\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-m74tl"
Oct 30 22:21:38.386164 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:38.386145    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e6f6850-1ca2-4f4c-a047-23e35ce2b697-istio-podinfo\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-m74tl\" (UID: \"3e6f6850-1ca2-4f4c-a047-23e35ce2b697\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-m74tl"
Oct 30 22:21:38.401481 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:38.401462    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e6f6850-1ca2-4f4c-a047-23e35ce2b697-istio-token\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-m74tl\" (UID: \"3e6f6850-1ca2-4f4c-a047-23e35ce2b697\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-m74tl"
Oct 30 22:21:38.401671 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:38.401653    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nfhf\" (UniqueName: \"kubernetes.io/projected/3e6f6850-1ca2-4f4c-a047-23e35ce2b697-kube-api-access-6nfhf\") pod \"e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-m74tl\" (UID: \"3e6f6850-1ca2-4f4c-a047-23e35ce2b697\") " pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-m74tl"
Oct 30 22:21:38.405145 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:38.405130    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-sqxvr"
Oct 30 22:21:38.413404 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:38.413382    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-m74tl"
Oct 30 22:21:38.550739 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:38.550708    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-9j5tr"]
Oct 30 22:21:38.554619 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:21:38.554593    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fb811da_4dc2_48bb_bec6_afb7a1a80ed2.slice/crio-d0da360d2cfb4d3c159d34274abb8f6d8fc4d2106af23a8fd9395d15a00462a8 WatchSource:0}: Error finding container d0da360d2cfb4d3c159d34274abb8f6d8fc4d2106af23a8fd9395d15a00462a8: Status 404 returned error can't find the container with id d0da360d2cfb4d3c159d34274abb8f6d8fc4d2106af23a8fd9395d15a00462a8
Oct 30 22:21:38.557009 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:38.556977    4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:21:38.557105 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:38.557075    4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:21:38.557162 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:38.557112    4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:21:38.709882 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:38.709858    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-sqxvr"]
Oct 30 22:21:38.713122 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:21:38.713036    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod909907e9_319a_48dc_90d6_735cee614217.slice/crio-68754249f74b3d328fb200ef5059eb3e8f621f8824d4cf09e5843a585e84c5db WatchSource:0}: Error finding container 68754249f74b3d328fb200ef5059eb3e8f621f8824d4cf09e5843a585e84c5db: Status 404 returned error can't find the container with id 68754249f74b3d328fb200ef5059eb3e8f621f8824d4cf09e5843a585e84c5db
Oct 30 22:21:38.715761 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:38.715718    4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:21:38.715854 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:38.715808    4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:21:38.715899 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:38.715874    4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:21:38.721082 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:38.721059    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-m74tl"]
Oct 30 22:21:38.726957 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:21:38.726933    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e6f6850_1ca2_4f4c_a047_23e35ce2b697.slice/crio-c008e4e77dd4e1d366bd28ddacdc81f83fd848320fb072983ab756d59d15f9ba WatchSource:0}: Error finding container c008e4e77dd4e1d366bd28ddacdc81f83fd848320fb072983ab756d59d15f9ba: Status 404 returned error can't find the container with id c008e4e77dd4e1d366bd28ddacdc81f83fd848320fb072983ab756d59d15f9ba
Oct 30 22:21:38.729819 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:38.729789    4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:21:38.729933 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:38.729846    4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:21:38.729933 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:38.729873    4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:21:39.254872 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:39.254834    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-9j5tr" event={"ID":"1fb811da-4dc2-48bb-bec6-afb7a1a80ed2","Type":"ContainerStarted","Data":"15834bc6d3f73decc1e21123036661a51b2cb8e7864cbf763806851838160302"}
Oct 30 22:21:39.255348 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:39.254897    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-9j5tr" event={"ID":"1fb811da-4dc2-48bb-bec6-afb7a1a80ed2","Type":"ContainerStarted","Data":"d0da360d2cfb4d3c159d34274abb8f6d8fc4d2106af23a8fd9395d15a00462a8"}
Oct 30 22:21:39.256601 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:39.256573    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-m74tl" event={"ID":"3e6f6850-1ca2-4f4c-a047-23e35ce2b697","Type":"ContainerStarted","Data":"ae78dc236996f12012a82d398789cf29f4474cf4a6bdc08ebf463df6b98553b3"}
Oct 30 22:21:39.256714 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:39.256621    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-m74tl" event={"ID":"3e6f6850-1ca2-4f4c-a047-23e35ce2b697","Type":"ContainerStarted","Data":"c008e4e77dd4e1d366bd28ddacdc81f83fd848320fb072983ab756d59d15f9ba"}
Oct 30 22:21:39.258064 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:39.258015    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-sqxvr" event={"ID":"909907e9-319a-48dc-90d6-735cee614217","Type":"ContainerStarted","Data":"5d137e93c191e9f87fea3d778f2115f94afccf75955bc716d7094b6c3717c143"}
Oct 30 22:21:39.258164 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:39.258070    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-sqxvr" event={"ID":"909907e9-319a-48dc-90d6-735cee614217","Type":"ContainerStarted","Data":"68754249f74b3d328fb200ef5059eb3e8f621f8824d4cf09e5843a585e84c5db"}
Oct 30 22:21:39.296731 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:39.296678    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-9j5tr" podStartSLOduration=2.296660137 podStartE2EDuration="2.296660137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:21:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:21:39.292868529 +0000 UTC m=+883.218794736" watchObservedRunningTime="2026-10-30 22:21:39.296660137 +0000 UTC m=+883.222586344"
Oct 30 22:21:39.340385 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:39.340312    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-sqxvr" podStartSLOduration=1.340290686 podStartE2EDuration="1.340290686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:21:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:21:39.332201498 +0000 UTC m=+883.258127705" watchObservedRunningTime="2026-10-30 22:21:39.340290686 +0000 UTC m=+883.266216892"
Oct 30 22:21:39.405997 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:39.405962    4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-sqxvr"
Oct 30 22:21:39.407804 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:39.407775    4477 patch_prober.go:29] interesting pod/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-sqxvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Oct 30 22:21:39.407954 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:39.407837    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-sqxvr" podUID="909907e9-319a-48dc-90d6-735cee614217" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Oct 30 22:21:39.413991 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:39.413965    4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-m74tl"
Oct 30 22:21:39.415126 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:39.415101    4477 patch_prober.go:29] interesting pod/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-m74tl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Oct 30 22:21:39.415220 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:39.415148    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-m74tl" podUID="3e6f6850-1ca2-4f4c-a047-23e35ce2b697" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Oct 30 22:21:40.241746 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:40.241703    4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-9j5tr"
Oct 30 22:21:40.244238 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:40.243910    4477 patch_prober.go:29] interesting pod/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-9j5tr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Oct 30 22:21:40.244238 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:40.243962    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-9j5tr" podUID="1fb811da-4dc2-48bb-bec6-afb7a1a80ed2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Oct 30 22:21:40.413065 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:40.412527    4477 patch_prober.go:29] interesting pod/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-sqxvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Oct 30 22:21:40.413065 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:40.412596    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-sqxvr" podUID="909907e9-319a-48dc-90d6-735cee614217" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Oct 30 22:21:40.440063 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:40.431294    4477 patch_prober.go:29] interesting pod/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-m74tl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Oct 30 22:21:40.440063 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:40.431367    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-m74tl" podUID="3e6f6850-1ca2-4f4c-a047-23e35ce2b697" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Oct 30 22:21:41.241560 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:41.241520    4477 patch_prober.go:29] interesting pod/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-9j5tr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Oct 30 22:21:41.241766 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:41.241584    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-9j5tr" podUID="1fb811da-4dc2-48bb-bec6-afb7a1a80ed2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Oct 30 22:21:41.410065 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:41.407067    4477 patch_prober.go:29] interesting pod/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-sqxvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Oct 30 22:21:41.410065 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:41.407123    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-sqxvr" podUID="909907e9-319a-48dc-90d6-735cee614217" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Oct 30 22:21:41.417072 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:41.416389    4477 patch_prober.go:29] interesting pod/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-m74tl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Oct 30 22:21:41.417072 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:41.416444    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-m74tl" podUID="3e6f6850-1ca2-4f4c-a047-23e35ce2b697" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Oct 30 22:21:42.244065 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:42.243365    4477 patch_prober.go:29] interesting pod/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-9j5tr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Oct 30 22:21:42.244065 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:42.243426    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-9j5tr" podUID="1fb811da-4dc2-48bb-bec6-afb7a1a80ed2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Oct 30 22:21:42.410541 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:42.410501    4477 patch_prober.go:29] interesting pod/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-sqxvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Oct 30 22:21:42.410819 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:42.410793    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-sqxvr" podUID="909907e9-319a-48dc-90d6-735cee614217" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Oct 30 22:21:42.417391 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:42.417364    4477 patch_prober.go:29] interesting pod/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-m74tl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Oct 30 22:21:42.417907 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:42.417882    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-m74tl" podUID="3e6f6850-1ca2-4f4c-a047-23e35ce2b697" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Oct 30 22:21:43.289180 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:43.289146    4477 patch_prober.go:29] interesting pod/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-9j5tr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Oct 30 22:21:43.296354 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:43.296315    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-9j5tr" podUID="1fb811da-4dc2-48bb-bec6-afb7a1a80ed2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Oct 30 22:21:43.413691 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:43.413656    4477 patch_prober.go:29] interesting pod/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-m74tl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Oct 30 22:21:43.413986 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:43.413957    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-m74tl" podUID="3e6f6850-1ca2-4f4c-a047-23e35ce2b697" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Oct 30 22:21:43.414588 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:43.414523    4477 patch_prober.go:29] interesting pod/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-sqxvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Oct 30 22:21:43.414748 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:43.414729    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-sqxvr" podUID="909907e9-319a-48dc-90d6-735cee614217" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Oct 30 22:21:44.251947 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:44.244146    4477 patch_prober.go:29] interesting pod/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-9j5tr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Oct 30 22:21:44.251947 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:44.244205    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-9j5tr" podUID="1fb811da-4dc2-48bb-bec6-afb7a1a80ed2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Oct 30 22:21:44.409539 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:44.409499    4477 patch_prober.go:29] interesting pod/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-sqxvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Oct 30 22:21:44.409813 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:44.409785    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-sqxvr" podUID="909907e9-319a-48dc-90d6-735cee614217" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Oct 30 22:21:44.414430 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:44.414401    4477 patch_prober.go:29] interesting pod/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-m74tl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Oct 30 22:21:44.414630 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:44.414606    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-m74tl" podUID="3e6f6850-1ca2-4f4c-a047-23e35ce2b697" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Oct 30 22:21:45.238401 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:45.238352    4477 patch_prober.go:29] interesting pod/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-9j5tr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Oct 30 22:21:45.238623 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:45.238416    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-9j5tr" podUID="1fb811da-4dc2-48bb-bec6-afb7a1a80ed2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Oct 30 22:21:45.405670 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:45.405631    4477 patch_prober.go:29] interesting pod/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-sqxvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Oct 30 22:21:45.406186 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:45.405703    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-sqxvr" podUID="909907e9-319a-48dc-90d6-735cee614217" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Oct 30 22:21:45.416032 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:45.416002    4477 patch_prober.go:29] interesting pod/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-m74tl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Oct 30 22:21:45.416288 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:45.416099    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-m74tl" podUID="3e6f6850-1ca2-4f4c-a047-23e35ce2b697" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Oct 30 22:21:46.646585 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:46.646542    4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-9j5tr"
Oct 30 22:21:46.647417 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:46.647398    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-9j5tr"
Oct 30 22:21:46.648325 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:46.648300    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-9j5tr"
Oct 30 22:21:46.699818 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:46.699712    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-m74tl" podStartSLOduration=8.699561561 podStartE2EDuration="8.699561561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:21:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:21:39.379652622 +0000 UTC m=+883.305578828" watchObservedRunningTime="2026-10-30 22:21:46.699561561 +0000 UTC m=+890.625487769"
Oct 30 22:21:46.805612 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:46.805574    4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-m74tl"
Oct 30 22:21:46.806029 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:46.806007    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-m74tl"
Oct 30 22:21:46.807171 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:46.807149    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-m74tl"
Oct 30 22:21:46.916778 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:46.916682    4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-sqxvr"
Oct 30 22:21:46.919232 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:46.919209    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-sqxvr"
Oct 30 22:21:46.924009 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:46.923986    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-sqxvr"
Oct 30 22:21:52.530768 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:52.530731    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-vtwp6"]
Oct 30 22:21:52.535123 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:52.535102    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-vtwp6"
Oct 30 22:21:52.565260 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:52.565237    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-z7ggc"]
Oct 30 22:21:52.569976 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:52.569954    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-vtwp6"]
Oct 30 22:21:52.570134 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:52.570094    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-z7ggc"
Oct 30 22:21:52.588538 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:52.588364    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-z7ggc"]
Oct 30 22:21:52.664774 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:52.664743    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5779d15a-20b4-4ec0-ac5e-b8bdfa89096d-istio-envoy\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-z7ggc\" (UID: \"5779d15a-20b4-4ec0-ac5e-b8bdfa89096d\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-z7ggc"
Oct 30 22:21:52.664774 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:52.664772    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/623d861e-ea33-4cd5-af9e-3b29b937e335-istio-envoy\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-vtwp6\" (UID: \"623d861e-ea33-4cd5-af9e-3b29b937e335\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-vtwp6"
Oct 30 22:21:52.664953 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:52.664791    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/623d861e-ea33-4cd5-af9e-3b29b937e335-istio-podinfo\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-vtwp6\" (UID: \"623d861e-ea33-4cd5-af9e-3b29b937e335\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-vtwp6"
Oct 30 22:21:52.664953 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:52.664807    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hd7cj\" (UniqueName: \"kubernetes.io/projected/623d861e-ea33-4cd5-af9e-3b29b937e335-kube-api-access-hd7cj\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-vtwp6\" (UID: \"623d861e-ea33-4cd5-af9e-3b29b937e335\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-vtwp6"
Oct 30 22:21:52.664953 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:52.664941    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5779d15a-20b4-4ec0-ac5e-b8bdfa89096d-workload-socket\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-z7ggc\" (UID: \"5779d15a-20b4-4ec0-ac5e-b8bdfa89096d\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-z7ggc"
Oct 30 22:21:52.665066 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:52.664979    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/623d861e-ea33-4cd5-af9e-3b29b937e335-workload-socket\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-vtwp6\" (UID: \"623d861e-ea33-4cd5-af9e-3b29b937e335\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-vtwp6"
Oct 30 22:21:52.665066 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:52.664998    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2cg9\" (UniqueName: \"kubernetes.io/projected/5779d15a-20b4-4ec0-ac5e-b8bdfa89096d-kube-api-access-l2cg9\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-z7ggc\" (UID: \"5779d15a-20b4-4ec0-ac5e-b8bdfa89096d\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-z7ggc"
Oct 30 22:21:52.665150 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:52.665131    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/623d861e-ea33-4cd5-af9e-3b29b937e335-istio-token\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-vtwp6\" (UID: \"623d861e-ea33-4cd5-af9e-3b29b937e335\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-vtwp6"
Oct 30 22:21:52.665183 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:52.665169    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5779d15a-20b4-4ec0-ac5e-b8bdfa89096d-credential-socket\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-z7ggc\" (UID: \"5779d15a-20b4-4ec0-ac5e-b8bdfa89096d\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-z7ggc"
Oct 30 22:21:52.665216 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:52.665192    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/623d861e-ea33-4cd5-af9e-3b29b937e335-credential-socket\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-vtwp6\" (UID: \"623d861e-ea33-4cd5-af9e-3b29b937e335\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-vtwp6"
Oct 30 22:21:52.665250 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:52.665240    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5779d15a-20b4-4ec0-ac5e-b8bdfa89096d-workload-certs\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-z7ggc\" (UID: \"5779d15a-20b4-4ec0-ac5e-b8bdfa89096d\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-z7ggc"
Oct 30 22:21:52.665292 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:52.665281    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5779d15a-20b4-4ec0-ac5e-b8bdfa89096d-istio-token\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-z7ggc\" (UID: \"5779d15a-20b4-4ec0-ac5e-b8bdfa89096d\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-z7ggc"
Oct 30 22:21:52.665337 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:52.665327    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/623d861e-ea33-4cd5-af9e-3b29b937e335-workload-certs\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-vtwp6\" (UID: \"623d861e-ea33-4cd5-af9e-3b29b937e335\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-vtwp6"
Oct 30 22:21:52.665368 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:52.665348    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/623d861e-ea33-4cd5-af9e-3b29b937e335-istio-data\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-vtwp6\" (UID: \"623d861e-ea33-4cd5-af9e-3b29b937e335\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-vtwp6"
Oct 30 22:21:52.665415 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:52.665402    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5779d15a-20b4-4ec0-ac5e-b8bdfa89096d-istiod-ca-cert\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-z7ggc\" (UID: \"5779d15a-20b4-4ec0-ac5e-b8bdfa89096d\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-z7ggc"
Oct 30 22:21:52.665449 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:52.665432    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5779d15a-20b4-4ec0-ac5e-b8bdfa89096d-istio-podinfo\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-z7ggc\" (UID: \"5779d15a-20b4-4ec0-ac5e-b8bdfa89096d\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-z7ggc"
Oct 30 22:21:52.665488 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:52.665451    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5779d15a-20b4-4ec0-ac5e-b8bdfa89096d-istio-data\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-z7ggc\" (UID: \"5779d15a-20b4-4ec0-ac5e-b8bdfa89096d\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-z7ggc"
Oct 30 22:21:52.665521 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:52.665498    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/623d861e-ea33-4cd5-af9e-3b29b937e335-istiod-ca-cert\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-vtwp6\" (UID: \"623d861e-ea33-4cd5-af9e-3b29b937e335\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-vtwp6"
Oct 30 22:21:52.766872 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:52.766842    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/623d861e-ea33-4cd5-af9e-3b29b937e335-istio-token\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-vtwp6\" (UID: \"623d861e-ea33-4cd5-af9e-3b29b937e335\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-vtwp6"
Oct 30 22:21:52.767013 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:52.766886    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5779d15a-20b4-4ec0-ac5e-b8bdfa89096d-credential-socket\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-z7ggc\" (UID: \"5779d15a-20b4-4ec0-ac5e-b8bdfa89096d\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-z7ggc"
Oct 30 22:21:52.767013 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:52.766918    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/623d861e-ea33-4cd5-af9e-3b29b937e335-credential-socket\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-vtwp6\" (UID: \"623d861e-ea33-4cd5-af9e-3b29b937e335\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-vtwp6"
Oct 30 22:21:52.767013 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:52.766947    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5779d15a-20b4-4ec0-ac5e-b8bdfa89096d-workload-certs\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-z7ggc\" (UID: \"5779d15a-20b4-4ec0-ac5e-b8bdfa89096d\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-z7ggc"
Oct 30 22:21:52.767013 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:52.766987    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5779d15a-20b4-4ec0-ac5e-b8bdfa89096d-istio-token\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-z7ggc\" (UID: \"5779d15a-20b4-4ec0-ac5e-b8bdfa89096d\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-z7ggc"
Oct 30 22:21:52.767269 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:52.767130    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/623d861e-ea33-4cd5-af9e-3b29b937e335-workload-certs\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-vtwp6\" (UID: \"623d861e-ea33-4cd5-af9e-3b29b937e335\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-vtwp6"
Oct 30 22:21:52.767269 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:52.767179    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/623d861e-ea33-4cd5-af9e-3b29b937e335-istio-data\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-vtwp6\" (UID: \"623d861e-ea33-4cd5-af9e-3b29b937e335\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-vtwp6"
Oct 30 22:21:52.767269 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:52.767236    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5779d15a-20b4-4ec0-ac5e-b8bdfa89096d-istiod-ca-cert\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-z7ggc\" (UID: \"5779d15a-20b4-4ec0-ac5e-b8bdfa89096d\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-z7ggc"
Oct 30 22:21:52.767431 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:52.767296    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5779d15a-20b4-4ec0-ac5e-b8bdfa89096d-istio-podinfo\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-z7ggc\" (UID: \"5779d15a-20b4-4ec0-ac5e-b8bdfa89096d\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-z7ggc"
Oct 30 22:21:52.767431 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:52.767320    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5779d15a-20b4-4ec0-ac5e-b8bdfa89096d-credential-socket\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-z7ggc\" (UID: \"5779d15a-20b4-4ec0-ac5e-b8bdfa89096d\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-z7ggc"
Oct 30 22:21:52.767431 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:52.767334    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5779d15a-20b4-4ec0-ac5e-b8bdfa89096d-istio-data\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-z7ggc\" (UID: \"5779d15a-20b4-4ec0-ac5e-b8bdfa89096d\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-z7ggc"
Oct 30 22:21:52.767431 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:52.767382    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/623d861e-ea33-4cd5-af9e-3b29b937e335-istiod-ca-cert\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-vtwp6\" (UID: \"623d861e-ea33-4cd5-af9e-3b29b937e335\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-vtwp6"
Oct 30 22:21:52.767431 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:52.767383    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5779d15a-20b4-4ec0-ac5e-b8bdfa89096d-workload-certs\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-z7ggc\" (UID: \"5779d15a-20b4-4ec0-ac5e-b8bdfa89096d\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-z7ggc"
Oct 30 22:21:52.767431 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:52.767387    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/623d861e-ea33-4cd5-af9e-3b29b937e335-credential-socket\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-vtwp6\" (UID: \"623d861e-ea33-4cd5-af9e-3b29b937e335\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-vtwp6"
Oct 30 22:21:52.767431 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:52.767417    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5779d15a-20b4-4ec0-ac5e-b8bdfa89096d-istio-envoy\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-z7ggc\" (UID: \"5779d15a-20b4-4ec0-ac5e-b8bdfa89096d\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-z7ggc"
Oct 30 22:21:52.767773 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:52.767442    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/623d861e-ea33-4cd5-af9e-3b29b937e335-istio-envoy\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-vtwp6\" (UID: \"623d861e-ea33-4cd5-af9e-3b29b937e335\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-vtwp6"
Oct 30 22:21:52.767773 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:52.767467    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/623d861e-ea33-4cd5-af9e-3b29b937e335-istio-podinfo\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-vtwp6\" (UID: \"623d861e-ea33-4cd5-af9e-3b29b937e335\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-vtwp6"
Oct 30 22:21:52.767773 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:52.767490    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hd7cj\" (UniqueName: \"kubernetes.io/projected/623d861e-ea33-4cd5-af9e-3b29b937e335-kube-api-access-hd7cj\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-vtwp6\" (UID: \"623d861e-ea33-4cd5-af9e-3b29b937e335\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-vtwp6"
Oct 30 22:21:52.767773 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:52.767469    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/623d861e-ea33-4cd5-af9e-3b29b937e335-workload-certs\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-vtwp6\" (UID: \"623d861e-ea33-4cd5-af9e-3b29b937e335\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-vtwp6"
Oct 30 22:21:52.767773 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:52.767583    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/623d861e-ea33-4cd5-af9e-3b29b937e335-istio-data\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-vtwp6\" (UID: \"623d861e-ea33-4cd5-af9e-3b29b937e335\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-vtwp6"
Oct 30 22:21:52.767773 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:52.767651    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5779d15a-20b4-4ec0-ac5e-b8bdfa89096d-istio-data\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-z7ggc\" (UID: \"5779d15a-20b4-4ec0-ac5e-b8bdfa89096d\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-z7ggc"
Oct 30 22:21:52.767773 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:52.767732    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5779d15a-20b4-4ec0-ac5e-b8bdfa89096d-workload-socket\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-z7ggc\" (UID: \"5779d15a-20b4-4ec0-ac5e-b8bdfa89096d\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-z7ggc"
Oct 30 22:21:52.768150 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:52.767792    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/623d861e-ea33-4cd5-af9e-3b29b937e335-workload-socket\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-vtwp6\" (UID: \"623d861e-ea33-4cd5-af9e-3b29b937e335\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-vtwp6"
Oct 30 22:21:52.768150 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:52.767814    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2cg9\" (UniqueName: \"kubernetes.io/projected/5779d15a-20b4-4ec0-ac5e-b8bdfa89096d-kube-api-access-l2cg9\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-z7ggc\" (UID: \"5779d15a-20b4-4ec0-ac5e-b8bdfa89096d\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-z7ggc"
Oct 30 22:21:52.768150 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:52.768008    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5779d15a-20b4-4ec0-ac5e-b8bdfa89096d-istiod-ca-cert\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-z7ggc\" (UID: \"5779d15a-20b4-4ec0-ac5e-b8bdfa89096d\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-z7ggc"
Oct 30 22:21:52.768310 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:52.768249    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/623d861e-ea33-4cd5-af9e-3b29b937e335-istiod-ca-cert\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-vtwp6\" (UID: \"623d861e-ea33-4cd5-af9e-3b29b937e335\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-vtwp6"
Oct 30 22:21:52.768310 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:52.768296    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/623d861e-ea33-4cd5-af9e-3b29b937e335-workload-socket\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-vtwp6\" (UID: \"623d861e-ea33-4cd5-af9e-3b29b937e335\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-vtwp6"
Oct 30 22:21:52.768450 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:52.768433    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5779d15a-20b4-4ec0-ac5e-b8bdfa89096d-workload-socket\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-z7ggc\" (UID: \"5779d15a-20b4-4ec0-ac5e-b8bdfa89096d\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-z7ggc"
Oct 30 22:21:52.769871 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:52.769852    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/623d861e-ea33-4cd5-af9e-3b29b937e335-istio-envoy\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-vtwp6\" (UID: \"623d861e-ea33-4cd5-af9e-3b29b937e335\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-vtwp6"
Oct 30 22:21:52.770318 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:52.770297    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5779d15a-20b4-4ec0-ac5e-b8bdfa89096d-istio-envoy\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-z7ggc\" (UID: \"5779d15a-20b4-4ec0-ac5e-b8bdfa89096d\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-z7ggc"
Oct 30 22:21:52.770497 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:52.770476    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5779d15a-20b4-4ec0-ac5e-b8bdfa89096d-istio-podinfo\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-z7ggc\" (UID: \"5779d15a-20b4-4ec0-ac5e-b8bdfa89096d\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-z7ggc"
Oct 30 22:21:52.770533 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:52.770481    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/623d861e-ea33-4cd5-af9e-3b29b937e335-istio-podinfo\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-vtwp6\" (UID: \"623d861e-ea33-4cd5-af9e-3b29b937e335\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-vtwp6"
Oct 30 22:21:52.787085 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:52.787019    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2cg9\" (UniqueName: \"kubernetes.io/projected/5779d15a-20b4-4ec0-ac5e-b8bdfa89096d-kube-api-access-l2cg9\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-z7ggc\" (UID: \"5779d15a-20b4-4ec0-ac5e-b8bdfa89096d\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-z7ggc"
Oct 30 22:21:52.787322 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:52.787299    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5779d15a-20b4-4ec0-ac5e-b8bdfa89096d-istio-token\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-z7ggc\" (UID: \"5779d15a-20b4-4ec0-ac5e-b8bdfa89096d\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-z7ggc"
Oct 30 22:21:52.787640 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:52.787621    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/623d861e-ea33-4cd5-af9e-3b29b937e335-istio-token\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-vtwp6\" (UID: \"623d861e-ea33-4cd5-af9e-3b29b937e335\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-vtwp6"
Oct 30 22:21:52.788009 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:52.787990    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hd7cj\" (UniqueName: \"kubernetes.io/projected/623d861e-ea33-4cd5-af9e-3b29b937e335-kube-api-access-hd7cj\") pod \"e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-vtwp6\" (UID: \"623d861e-ea33-4cd5-af9e-3b29b937e335\") " pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-vtwp6"
Oct 30 22:21:52.849917 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:52.849888    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-vtwp6"
Oct 30 22:21:52.887668 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:52.887643    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-z7ggc"
Oct 30 22:21:53.170362 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:53.170339    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-vtwp6"]
Oct 30 22:21:53.173758 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:21:53.173732    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod623d861e_ea33_4cd5_af9e_3b29b937e335.slice/crio-962ead7ef6210c04a01ce515730436cadc44075e5ecd42658435224cddfb6862 WatchSource:0}: Error finding container 962ead7ef6210c04a01ce515730436cadc44075e5ecd42658435224cddfb6862: Status 404 returned error can't find the container with id 962ead7ef6210c04a01ce515730436cadc44075e5ecd42658435224cddfb6862
Oct 30 22:21:53.176170 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:53.176137    4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:21:53.176237 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:53.176201    4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:21:53.176237 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:53.176228    4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:21:53.216701 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:53.216668    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-z7ggc"]
Oct 30 22:21:53.221889 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:21:53.221858    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5779d15a_20b4_4ec0_ac5e_b8bdfa89096d.slice/crio-22cd51e1b69325ca20e47ad6a2d590e33e9d46ef006e0cd1dde5ad480bbc15ba WatchSource:0}: Error finding container 22cd51e1b69325ca20e47ad6a2d590e33e9d46ef006e0cd1dde5ad480bbc15ba: Status 404 returned error can't find the container with id 22cd51e1b69325ca20e47ad6a2d590e33e9d46ef006e0cd1dde5ad480bbc15ba
Oct 30 22:21:53.224324 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:53.224293    4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:21:53.224395 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:53.224355    4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:21:53.224395 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:53.224384    4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:21:53.396624 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:53.396588    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-spjhs"]
Oct 30 22:21:53.400899 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:53.400878    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-spjhs"
Oct 30 22:21:53.414233 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:53.414212    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-421583-openshift-default-dockercfg-fp7vh\""
Oct 30 22:21:53.423208 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:53.423138    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-spjhs"]
Oct 30 22:21:53.454560 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:53.454529    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-vtwp6" event={"ID":"623d861e-ea33-4cd5-af9e-3b29b937e335","Type":"ContainerStarted","Data":"be0bd308f5ff525861432f5dc7749174652c7a6e2bfaa27dda66b70eeffd202a"}
Oct 30 22:21:53.454701 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:53.454582    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-vtwp6" event={"ID":"623d861e-ea33-4cd5-af9e-3b29b937e335","Type":"ContainerStarted","Data":"962ead7ef6210c04a01ce515730436cadc44075e5ecd42658435224cddfb6862"}
Oct 30 22:21:53.455965 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:53.455939    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-z7ggc" event={"ID":"5779d15a-20b4-4ec0-ac5e-b8bdfa89096d","Type":"ContainerStarted","Data":"2444cbe557ac681373e56ac69c36d693830c7a572f5cd296ec1b7c95dbaebe32"}
Oct 30 22:21:53.456097 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:53.455986    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-z7ggc" event={"ID":"5779d15a-20b4-4ec0-ac5e-b8bdfa89096d","Type":"ContainerStarted","Data":"22cd51e1b69325ca20e47ad6a2d590e33e9d46ef006e0cd1dde5ad480bbc15ba"}
Oct 30 22:21:53.477028 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:53.476991    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e1402af-909e-4800-953a-5b92138194dc-istio-envoy\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-spjhs\" (UID: \"5e1402af-909e-4800-953a-5b92138194dc\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-spjhs"
Oct 30 22:21:53.477171 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:53.477146    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e1402af-909e-4800-953a-5b92138194dc-istio-token\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-spjhs\" (UID: \"5e1402af-909e-4800-953a-5b92138194dc\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-spjhs"
Oct 30 22:21:53.477281 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:53.477260    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e1402af-909e-4800-953a-5b92138194dc-credential-socket\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-spjhs\" (UID: \"5e1402af-909e-4800-953a-5b92138194dc\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-spjhs"
Oct 30 22:21:53.477343 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:53.477297    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e1402af-909e-4800-953a-5b92138194dc-istio-data\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-spjhs\" (UID: \"5e1402af-909e-4800-953a-5b92138194dc\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-spjhs"
Oct 30 22:21:53.477455 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:53.477432    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e1402af-909e-4800-953a-5b92138194dc-workload-certs\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-spjhs\" (UID: \"5e1402af-909e-4800-953a-5b92138194dc\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-spjhs"
Oct 30 22:21:53.477574 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:53.477558    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e1402af-909e-4800-953a-5b92138194dc-istio-podinfo\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-spjhs\" (UID: \"5e1402af-909e-4800-953a-5b92138194dc\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-spjhs"
Oct 30 22:21:53.477629 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:53.477616    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e1402af-909e-4800-953a-5b92138194dc-istiod-ca-cert\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-spjhs\" (UID: \"5e1402af-909e-4800-953a-5b92138194dc\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-spjhs"
Oct 30 22:21:53.477708 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:53.477688    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hqzk\" (UniqueName: \"kubernetes.io/projected/5e1402af-909e-4800-953a-5b92138194dc-kube-api-access-8hqzk\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-spjhs\" (UID: \"5e1402af-909e-4800-953a-5b92138194dc\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-spjhs"
Oct 30 22:21:53.477782 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:53.477767    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e1402af-909e-4800-953a-5b92138194dc-workload-socket\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-spjhs\" (UID: \"5e1402af-909e-4800-953a-5b92138194dc\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-spjhs"
Oct 30 22:21:53.495963 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:53.495909    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-vtwp6" podStartSLOduration=1.495893429 podStartE2EDuration="1.495893429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:21:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:21:53.495283084 +0000 UTC m=+897.421209313" watchObservedRunningTime="2026-10-30 22:21:53.495893429 +0000 UTC m=+897.421819635"
Oct 30 22:21:53.541905 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:53.541857    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-z7ggc" podStartSLOduration=1.541843452 podStartE2EDuration="1.541843452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:21:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:21:53.537699167 +0000 UTC m=+897.463625416" watchObservedRunningTime="2026-10-30 22:21:53.541843452 +0000 UTC m=+897.467769705"
Oct 30 22:21:53.579377 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:53.579344    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e1402af-909e-4800-953a-5b92138194dc-istio-envoy\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-spjhs\" (UID: \"5e1402af-909e-4800-953a-5b92138194dc\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-spjhs"
Oct 30 22:21:53.579551 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:53.579443    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e1402af-909e-4800-953a-5b92138194dc-istio-token\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-spjhs\" (UID: \"5e1402af-909e-4800-953a-5b92138194dc\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-spjhs"
Oct 30 22:21:53.579665 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:53.579635    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e1402af-909e-4800-953a-5b92138194dc-credential-socket\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-spjhs\" (UID: \"5e1402af-909e-4800-953a-5b92138194dc\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-spjhs"
Oct 30 22:21:53.579721 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:53.579681    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e1402af-909e-4800-953a-5b92138194dc-istio-data\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-spjhs\" (UID: \"5e1402af-909e-4800-953a-5b92138194dc\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-spjhs"
Oct 30 22:21:53.579779 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:53.579769    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e1402af-909e-4800-953a-5b92138194dc-workload-certs\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-spjhs\" (UID: \"5e1402af-909e-4800-953a-5b92138194dc\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-spjhs"
Oct 30 22:21:53.579962 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:53.579937    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e1402af-909e-4800-953a-5b92138194dc-istio-podinfo\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-spjhs\" (UID: \"5e1402af-909e-4800-953a-5b92138194dc\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-spjhs"
Oct 30 22:21:53.580102 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:53.580080    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e1402af-909e-4800-953a-5b92138194dc-credential-socket\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-spjhs\" (UID: \"5e1402af-909e-4800-953a-5b92138194dc\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-spjhs"
Oct 30 22:21:53.580160 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:53.580108    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e1402af-909e-4800-953a-5b92138194dc-istiod-ca-cert\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-spjhs\" (UID: \"5e1402af-909e-4800-953a-5b92138194dc\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-spjhs"
Oct 30 22:21:53.580160 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:53.580142    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hqzk\" (UniqueName: \"kubernetes.io/projected/5e1402af-909e-4800-953a-5b92138194dc-kube-api-access-8hqzk\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-spjhs\" (UID: \"5e1402af-909e-4800-953a-5b92138194dc\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-spjhs"
Oct 30 22:21:53.580276 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:53.580175    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e1402af-909e-4800-953a-5b92138194dc-istio-data\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-spjhs\" (UID: \"5e1402af-909e-4800-953a-5b92138194dc\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-spjhs"
Oct 30 22:21:53.580625 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:53.580599    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e1402af-909e-4800-953a-5b92138194dc-workload-socket\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-spjhs\" (UID: \"5e1402af-909e-4800-953a-5b92138194dc\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-spjhs"
Oct 30 22:21:53.581153 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:53.581111    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e1402af-909e-4800-953a-5b92138194dc-workload-socket\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-spjhs\" (UID: \"5e1402af-909e-4800-953a-5b92138194dc\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-spjhs"
Oct 30 22:21:53.581278 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:53.581163    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e1402af-909e-4800-953a-5b92138194dc-workload-certs\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-spjhs\" (UID: \"5e1402af-909e-4800-953a-5b92138194dc\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-spjhs"
Oct 30 22:21:53.581490 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:53.581466    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e1402af-909e-4800-953a-5b92138194dc-istiod-ca-cert\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-spjhs\" (UID: \"5e1402af-909e-4800-953a-5b92138194dc\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-spjhs"
Oct 30 22:21:53.583036 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:53.583011    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e1402af-909e-4800-953a-5b92138194dc-istio-podinfo\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-spjhs\" (UID: \"5e1402af-909e-4800-953a-5b92138194dc\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-spjhs"
Oct 30 22:21:53.583418 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:53.583395    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e1402af-909e-4800-953a-5b92138194dc-istio-envoy\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-spjhs\" (UID: \"5e1402af-909e-4800-953a-5b92138194dc\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-spjhs"
Oct 30 22:21:53.598524 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:53.598501    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e1402af-909e-4800-953a-5b92138194dc-istio-token\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-spjhs\" (UID: \"5e1402af-909e-4800-953a-5b92138194dc\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-spjhs"
Oct 30 22:21:53.598661 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:53.598640    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hqzk\" (UniqueName: \"kubernetes.io/projected/5e1402af-909e-4800-953a-5b92138194dc-kube-api-access-8hqzk\") pod \"e2e-worker-w2-421583-openshift-default-5cb767b447-spjhs\" (UID: \"5e1402af-909e-4800-953a-5b92138194dc\") " pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-spjhs"
Oct 30 22:21:53.715994 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:53.715924    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-spjhs"
Oct 30 22:21:53.850207 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:53.850180    4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-vtwp6"
Oct 30 22:21:53.888753 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:53.888731    4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-z7ggc"
Oct 30 22:21:54.035120 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:54.033661    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-spjhs"]
Oct 30 22:21:54.042015 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:21:54.041988    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e1402af_909e_4800_953a_5b92138194dc.slice/crio-e1a3ce4334135ac88c367ba88b23c025e085a16194ae3ea403b127cd5b97f910 WatchSource:0}: Error finding container e1a3ce4334135ac88c367ba88b23c025e085a16194ae3ea403b127cd5b97f910: Status 404 returned error can't find the container with id e1a3ce4334135ac88c367ba88b23c025e085a16194ae3ea403b127cd5b97f910
Oct 30 22:21:54.045269 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:54.045226    4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:21:54.045339 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:54.045319    4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:21:54.045383 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:54.045362    4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:21:54.522065 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:54.503195    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-spjhs" event={"ID":"5e1402af-909e-4800-953a-5b92138194dc","Type":"ContainerStarted","Data":"58fb6b84e0a11aeda8513e1ff24b66a60c058eb06a4ed70fbe70310dd0fbac1d"}
Oct 30 22:21:54.522065 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:54.503253    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-spjhs" event={"ID":"5e1402af-909e-4800-953a-5b92138194dc","Type":"ContainerStarted","Data":"e1a3ce4334135ac88c367ba88b23c025e085a16194ae3ea403b127cd5b97f910"}
Oct 30 22:21:54.558381 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:54.558321    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-spjhs" podStartSLOduration=1.558308156 podStartE2EDuration="1.558308156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:21:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:21:54.556814418 +0000 UTC m=+898.482740623" watchObservedRunningTime="2026-10-30 22:21:54.558308156 +0000 UTC m=+898.484234361"
Oct 30 22:21:54.718063 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:54.716139    4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-spjhs"
Oct 30 22:21:54.852214 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:54.852134    4477 patch_prober.go:29] interesting pod/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-vtwp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Oct 30 22:21:54.852214 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:54.852201    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-vtwp6" podUID="623d861e-ea33-4cd5-af9e-3b29b937e335" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Oct 30 22:21:54.896193 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:54.895859    4477 patch_prober.go:29] interesting pod/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-z7ggc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Oct 30 22:21:54.896193 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:54.895920    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-z7ggc" podUID="5779d15a-20b4-4ec0-ac5e-b8bdfa89096d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Oct 30 22:21:55.718229 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:55.718190    4477 patch_prober.go:29] interesting pod/e2e-worker-w2-421583-openshift-default-5cb767b447-spjhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Oct 30 22:21:55.718898 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:55.718861    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-spjhs" podUID="5e1402af-909e-4800-953a-5b92138194dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Oct 30 22:21:55.854860 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:55.854410    4477 patch_prober.go:29] interesting pod/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-vtwp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Oct 30 22:21:55.854860 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:55.854473    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-vtwp6" podUID="623d861e-ea33-4cd5-af9e-3b29b937e335" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Oct 30 22:21:55.892667 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:55.892632    4477 patch_prober.go:29] interesting pod/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-z7ggc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Oct 30 22:21:55.892933 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:55.892909    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-z7ggc" podUID="5779d15a-20b4-4ec0-ac5e-b8bdfa89096d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Oct 30 22:21:56.552067 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:21:56.550685    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 22:21:56.597060 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:56.586144    4477 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jcf8l_015f218d-bf1f-4e7e-bb5d-b805af003246/ovn-acl-logging/0.log"
Oct 30 22:21:56.718066 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:56.717646    4477 patch_prober.go:29] interesting pod/e2e-worker-w2-421583-openshift-default-5cb767b447-spjhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Oct 30 22:21:56.718066 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:56.717703    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-spjhs" podUID="5e1402af-909e-4800-953a-5b92138194dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Oct 30 22:21:56.758073 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:56.754616    4477 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jcf8l_015f218d-bf1f-4e7e-bb5d-b805af003246/ovn-acl-logging/0.log"
Oct 30 22:21:56.857745 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:56.857654    4477 patch_prober.go:29] interesting pod/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-vtwp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Oct 30 22:21:56.857994 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:56.857965    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-vtwp6" podUID="623d861e-ea33-4cd5-af9e-3b29b937e335" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Oct 30 22:21:56.890410 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:56.890356    4477 patch_prober.go:29] interesting pod/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-z7ggc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Oct 30 22:21:56.890581 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:56.890413    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-z7ggc" podUID="5779d15a-20b4-4ec0-ac5e-b8bdfa89096d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Oct 30 22:21:57.723062 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:57.722391    4477 patch_prober.go:29] interesting pod/e2e-worker-w2-421583-openshift-default-5cb767b447-spjhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Oct 30 22:21:57.723062 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:57.722451    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-spjhs" podUID="5e1402af-909e-4800-953a-5b92138194dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Oct 30 22:21:57.855064 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:57.851366    4477 patch_prober.go:29] interesting pod/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-vtwp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Oct 30 22:21:57.855064 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:57.851440    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-vtwp6" podUID="623d861e-ea33-4cd5-af9e-3b29b937e335" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Oct 30 22:21:57.892069 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:57.890540    4477 patch_prober.go:29] interesting pod/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-z7ggc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Oct 30 22:21:57.892069 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:57.890594    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-z7ggc" podUID="5779d15a-20b4-4ec0-ac5e-b8bdfa89096d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Oct 30 22:21:58.720068 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:58.717427    4477 patch_prober.go:29] interesting pod/e2e-worker-w2-421583-openshift-default-5cb767b447-spjhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Oct 30 22:21:58.720068 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:58.717486    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-spjhs" podUID="5e1402af-909e-4800-953a-5b92138194dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Oct 30 22:21:58.850321 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:58.850281    4477 patch_prober.go:29] interesting pod/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-vtwp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Oct 30 22:21:58.850716 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:58.850347    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-vtwp6" podUID="623d861e-ea33-4cd5-af9e-3b29b937e335" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Oct 30 22:21:58.901065 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:58.893172    4477 patch_prober.go:29] interesting pod/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-z7ggc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Oct 30 22:21:58.901065 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:58.893247    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-z7ggc" podUID="5779d15a-20b4-4ec0-ac5e-b8bdfa89096d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Oct 30 22:21:59.716794 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:59.716755    4477 patch_prober.go:29] interesting pod/e2e-worker-w2-421583-openshift-default-5cb767b447-spjhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Oct 30 22:21:59.716980 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:59.716817    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-spjhs" podUID="5e1402af-909e-4800-953a-5b92138194dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Oct 30 22:21:59.789063 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:59.788955    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6bz92"]
Oct 30 22:21:59.850253 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:59.850214    4477 patch_prober.go:29] interesting pod/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-vtwp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Oct 30 22:21:59.850409 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:59.850283    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-vtwp6" podUID="623d861e-ea33-4cd5-af9e-3b29b937e335" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Oct 30 22:21:59.890071 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:59.889472    4477 patch_prober.go:29] interesting pod/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-z7ggc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Oct 30 22:21:59.890071 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:21:59.889544    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-z7ggc" podUID="5779d15a-20b4-4ec0-ac5e-b8bdfa89096d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Oct 30 22:22:00.732065 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:00.722785    4477 patch_prober.go:29] interesting pod/e2e-worker-w2-421583-openshift-default-5cb767b447-spjhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Oct 30 22:22:00.732065 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:00.722845    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-spjhs" podUID="5e1402af-909e-4800-953a-5b92138194dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Oct 30 22:22:01.723066 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:01.722244    4477 patch_prober.go:29] interesting pod/e2e-worker-w2-421583-openshift-default-5cb767b447-spjhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Oct 30 22:22:01.723066 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:01.722325    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-spjhs" podUID="5e1402af-909e-4800-953a-5b92138194dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Oct 30 22:22:01.770095 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:01.769432    4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-z7ggc"
Oct 30 22:22:01.771785 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:01.771024    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-z7ggc"
Oct 30 22:22:01.785435 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:01.785366    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-z7ggc"
Oct 30 22:22:01.863155 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:01.863117    4477 patch_prober.go:29] interesting pod/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-vtwp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:22:01.863437 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:01.863403    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-vtwp6" podUID="623d861e-ea33-4cd5-af9e-3b29b937e335" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:01.957742 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:01.957707    4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-vtwp6"
Oct 30 22:22:01.996113 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:01.970830    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-vtwp6"
Oct 30 22:22:02.026063 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:02.000550    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-vtwp6"
Oct 30 22:22:03.302066 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:03.296930    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/simulator-w1-2b1e88-kserve-6c579b99f8-q5wpq" podUID="722f10d8-b7a2-4bf7-a274-9669529ef427" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:03.786069 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:03.776160    4477 patch_prober.go:29] interesting pod/e2e-worker-w2-421583-openshift-default-5cb767b447-spjhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:22:03.786069 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:03.776274    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-spjhs" podUID="5e1402af-909e-4800-953a-5b92138194dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:03.861066 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:03.858117    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0ad7c5/premium-w4-0ad7c5-kserve-7bbf78cd4-pm6wc" podUID="5c10371e-56e7-42ec-9ef8-0802e15cb2ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:03.938215 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:03.938170    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/distinct-2-w6-012ba0-kserve-78dbcd8c46-hqdfm" podUID="8326b014-989b-4722-befb-6fa8ea60688b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Oct 30 22:22:03.940361 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:03.938461    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/distinct-w3-d52c59-kserve-ffb5f5b96-l58xl" podUID="c7b1483b-2631-4aa2-a888-3096761d5494" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:03.972657 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:03.970162    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-2-w4-0ad7c5-kserve-77c5467fcb-skspk" podUID="430bd879-2a89-4070-bb1b-9b9958a261b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Oct 30 22:22:04.097238 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:04.097142    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/distinct-2-w0-09d0d5-kserve-7dcf89b98b-7cjph" podUID="8cbd04b4-bb3e-4bbb-a41a-46131c9de051" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:04.303081 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:04.303016    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/simulator-w1-2b1e88-kserve-6c579b99f8-q5wpq" podUID="722f10d8-b7a2-4bf7-a274-9669529ef427" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:04.313071 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:04.313022    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-2b1e88/simulator-w1-2b1e88-kserve-6c579b99f8-q5wpq"
Oct 30 22:22:04.489768 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:04.489728    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/premium-w6-012ba0-kserve-658654dbdb-d2k27" podUID="085847e5-4f50-419e-984f-b612b104ed90" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded"
Oct 30 22:22:04.526064 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:04.515635    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-2b1e88/simulator-w1-2b1e88-kserve-6c579b99f8-q5wpq"
Oct 30 22:22:04.836120 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:04.832296    4477 patch_prober.go:29] interesting pod/e2e-worker-w2-421583-openshift-default-5cb767b447-spjhs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:22:04.836120 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:04.832378    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-spjhs" podUID="5e1402af-909e-4800-953a-5b92138194dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:04.857152 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:04.857111    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/distinct-w2-421583-kserve-69cf5f4d6b-qbgnt" podUID="510ad10d-0554-47c8-9b55-deb359795071" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:04.871680 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:04.871262    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0ad7c5/premium-w4-0ad7c5-kserve-7bbf78cd4-pm6wc" podUID="5c10371e-56e7-42ec-9ef8-0802e15cb2ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:04.873391 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:04.871930    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0ad7c5/premium-w4-0ad7c5-kserve-7bbf78cd4-pm6wc"
Oct 30 22:22:05.009211 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:05.009174    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-2-w4-0ad7c5-kserve-77c5467fcb-skspk" podUID="430bd879-2a89-4070-bb1b-9b9958a261b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:05.043133 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:05.010259    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-2-w4-0ad7c5-kserve-77c5467fcb-skspk"
Oct 30 22:22:05.047161 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:05.045140    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/premium-w5-54114e-kserve-57f6747bcc-7wjrw" podUID="61990d22-ab8f-45bc-a4d6-4c8d528742f1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:05.455199 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:05.452688    4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-spjhs"
Oct 30 22:22:05.462210 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:05.462030    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-spjhs"
Oct 30 22:22:05.470920 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:05.470780    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-spjhs"
Oct 30 22:22:05.520847 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:05.520804    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/simulator-w1-2b1e88-kserve-6c579b99f8-q5wpq" podUID="722f10d8-b7a2-4bf7-a274-9669529ef427" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:05.660066 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:05.657887    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0ad7c5/premium-w4-0ad7c5-kserve-7bbf78cd4-pm6wc"
Oct 30 22:22:06.014064 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:06.012095    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-2-w4-0ad7c5-kserve-77c5467fcb-skspk" podUID="430bd879-2a89-4070-bb1b-9b9958a261b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Oct 30 22:22:06.151898 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:06.151874    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-2-w4-0ad7c5-kserve-77c5467fcb-skspk"
Oct 30 22:22:23.460561 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:23.460502    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-crp67"]
Oct 30 22:22:23.542066 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:23.516487    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-crp67"
Oct 30 22:22:23.572991 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:23.553118    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/simulator-w1-2b1e88-kserve-6c579b99f8-q5wpq" podUID="722f10d8-b7a2-4bf7-a274-9669529ef427" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:23.845144 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:23.844185    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f5f8741-f496-46a2-b357-72c0e27ea2ae-workload-certs\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-crp67\" (UID: \"2f5f8741-f496-46a2-b357-72c0e27ea2ae\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-crp67"
Oct 30 22:22:23.845144 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:23.844274    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f5f8741-f496-46a2-b357-72c0e27ea2ae-istio-token\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-crp67\" (UID: \"2f5f8741-f496-46a2-b357-72c0e27ea2ae\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-crp67"
Oct 30 22:22:23.845144 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:23.844448    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f5f8741-f496-46a2-b357-72c0e27ea2ae-credential-socket\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-crp67\" (UID: \"2f5f8741-f496-46a2-b357-72c0e27ea2ae\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-crp67"
Oct 30 22:22:23.845144 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:23.844650    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f5f8741-f496-46a2-b357-72c0e27ea2ae-workload-socket\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-crp67\" (UID: \"2f5f8741-f496-46a2-b357-72c0e27ea2ae\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-crp67"
Oct 30 22:22:23.845144 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:23.844686    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f5f8741-f496-46a2-b357-72c0e27ea2ae-istio-envoy\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-crp67\" (UID: \"2f5f8741-f496-46a2-b357-72c0e27ea2ae\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-crp67"
Oct 30 22:22:23.845144 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:23.844717    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f5f8741-f496-46a2-b357-72c0e27ea2ae-istio-data\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-crp67\" (UID: \"2f5f8741-f496-46a2-b357-72c0e27ea2ae\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-crp67"
Oct 30 22:22:23.845144 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:23.844756    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7chh\" (UniqueName: \"kubernetes.io/projected/2f5f8741-f496-46a2-b357-72c0e27ea2ae-kube-api-access-k7chh\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-crp67\" (UID: \"2f5f8741-f496-46a2-b357-72c0e27ea2ae\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-crp67"
Oct 30 22:22:23.845144 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:23.844958    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f5f8741-f496-46a2-b357-72c0e27ea2ae-istiod-ca-cert\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-crp67\" (UID: \"2f5f8741-f496-46a2-b357-72c0e27ea2ae\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-crp67"
Oct 30 22:22:23.845144 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:23.845002    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f5f8741-f496-46a2-b357-72c0e27ea2ae-istio-podinfo\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-crp67\" (UID: \"2f5f8741-f496-46a2-b357-72c0e27ea2ae\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-crp67"
Oct 30 22:22:23.865097 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:23.858637    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/distinct-w2-421583-kserve-69cf5f4d6b-qbgnt" podUID="510ad10d-0554-47c8-9b55-deb359795071" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:23.919228 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:23.919182    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/distinct-w3-d52c59-kserve-ffb5f5b96-l58xl" podUID="c7b1483b-2631-4aa2-a888-3096761d5494" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:23.944844 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:23.944807    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/distinct-2-w6-012ba0-kserve-78dbcd8c46-hqdfm" podUID="8326b014-989b-4722-befb-6fa8ea60688b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:23.945787 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:23.945503    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-crp67"]
Oct 30 22:22:23.984068 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:23.973844    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f5f8741-f496-46a2-b357-72c0e27ea2ae-workload-certs\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-crp67\" (UID: \"2f5f8741-f496-46a2-b357-72c0e27ea2ae\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-crp67"
Oct 30 22:22:23.986807 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:23.986779    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f5f8741-f496-46a2-b357-72c0e27ea2ae-istio-token\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-crp67\" (UID: \"2f5f8741-f496-46a2-b357-72c0e27ea2ae\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-crp67"
Oct 30 22:22:24.003369 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:24.003321    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f5f8741-f496-46a2-b357-72c0e27ea2ae-credential-socket\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-crp67\" (UID: \"2f5f8741-f496-46a2-b357-72c0e27ea2ae\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-crp67"
Oct 30 22:22:24.003530 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:24.003416    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f5f8741-f496-46a2-b357-72c0e27ea2ae-workload-socket\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-crp67\" (UID: \"2f5f8741-f496-46a2-b357-72c0e27ea2ae\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-crp67"
Oct 30 22:22:24.003530 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:24.003458    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f5f8741-f496-46a2-b357-72c0e27ea2ae-istio-envoy\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-crp67\" (UID: \"2f5f8741-f496-46a2-b357-72c0e27ea2ae\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-crp67"
Oct 30 22:22:24.003530 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:24.003497    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f5f8741-f496-46a2-b357-72c0e27ea2ae-istio-data\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-crp67\" (UID: \"2f5f8741-f496-46a2-b357-72c0e27ea2ae\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-crp67"
Oct 30 22:22:24.003694 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:24.003535    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7chh\" (UniqueName: \"kubernetes.io/projected/2f5f8741-f496-46a2-b357-72c0e27ea2ae-kube-api-access-k7chh\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-crp67\" (UID: \"2f5f8741-f496-46a2-b357-72c0e27ea2ae\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-crp67"
Oct 30 22:22:24.003694 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:24.003585    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f5f8741-f496-46a2-b357-72c0e27ea2ae-istiod-ca-cert\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-crp67\" (UID: \"2f5f8741-f496-46a2-b357-72c0e27ea2ae\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-crp67"
Oct 30 22:22:24.003694 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:24.003631    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f5f8741-f496-46a2-b357-72c0e27ea2ae-istio-podinfo\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-crp67\" (UID: \"2f5f8741-f496-46a2-b357-72c0e27ea2ae\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-crp67"
Oct 30 22:22:24.025123 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:23.986909    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f5f8741-f496-46a2-b357-72c0e27ea2ae-workload-certs\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-crp67\" (UID: \"2f5f8741-f496-46a2-b357-72c0e27ea2ae\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-crp67"
Oct 30 22:22:24.026671 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:24.025799    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f5f8741-f496-46a2-b357-72c0e27ea2ae-credential-socket\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-crp67\" (UID: \"2f5f8741-f496-46a2-b357-72c0e27ea2ae\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-crp67"
Oct 30 22:22:24.054060 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:24.001965    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/premium-w5-54114e-kserve-57f6747bcc-7wjrw" podUID="61990d22-ab8f-45bc-a4d6-4c8d528742f1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded"
Oct 30 22:22:24.054660 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:24.026096    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f5f8741-f496-46a2-b357-72c0e27ea2ae-workload-socket\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-crp67\" (UID: \"2f5f8741-f496-46a2-b357-72c0e27ea2ae\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-crp67"
Oct 30 22:22:24.054781 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:24.027669    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f5f8741-f496-46a2-b357-72c0e27ea2ae-istio-data\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-crp67\" (UID: \"2f5f8741-f496-46a2-b357-72c0e27ea2ae\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-crp67"
Oct 30 22:22:24.054781 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:24.028435    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f5f8741-f496-46a2-b357-72c0e27ea2ae-istiod-ca-cert\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-crp67\" (UID: \"2f5f8741-f496-46a2-b357-72c0e27ea2ae\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-crp67"
Oct 30 22:22:24.054781 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:24.028498    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-2-w4-0ad7c5-kserve-77c5467fcb-skspk" podUID="430bd879-2a89-4070-bb1b-9b9958a261b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:24.067063 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:24.039478    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f5f8741-f496-46a2-b357-72c0e27ea2ae-istio-envoy\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-crp67\" (UID: \"2f5f8741-f496-46a2-b357-72c0e27ea2ae\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-crp67"
Oct 30 22:22:24.067063 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:24.049408    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/distinct-2-w0-09d0d5-kserve-7dcf89b98b-7cjph" podUID="8cbd04b4-bb3e-4bbb-a41a-46131c9de051" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:24.073063 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:24.069609    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f5f8741-f496-46a2-b357-72c0e27ea2ae-istio-token\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-crp67\" (UID: \"2f5f8741-f496-46a2-b357-72c0e27ea2ae\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-crp67"
Oct 30 22:22:24.077128 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:24.073864    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f5f8741-f496-46a2-b357-72c0e27ea2ae-istio-podinfo\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-crp67\" (UID: \"2f5f8741-f496-46a2-b357-72c0e27ea2ae\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-crp67"
Oct 30 22:22:24.085109 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:24.083712    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7chh\" (UniqueName: \"kubernetes.io/projected/2f5f8741-f496-46a2-b357-72c0e27ea2ae-kube-api-access-k7chh\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-crp67\" (UID: \"2f5f8741-f496-46a2-b357-72c0e27ea2ae\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-crp67"
Oct 30 22:22:24.130111 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:24.111241    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-crp67"
Oct 30 22:22:24.695097 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:24.632867    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/premium-w6-012ba0-kserve-658654dbdb-d2k27" podUID="085847e5-4f50-419e-984f-b612b104ed90" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:24.748555 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:24.640224    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/simulator-w1-2b1e88-kserve-6c579b99f8-q5wpq" podUID="722f10d8-b7a2-4bf7-a274-9669529ef427" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:24.756396 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:24.756350    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-2b1e88/simulator-w1-2b1e88-kserve-6c579b99f8-q5wpq"
Oct 30 22:22:24.857505 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:24.857453    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0ad7c5/premium-w4-0ad7c5-kserve-7bbf78cd4-pm6wc" podUID="5c10371e-56e7-42ec-9ef8-0802e15cb2ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded"
Oct 30 22:22:24.917707 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:24.917666    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/distinct-w2-421583-kserve-69cf5f4d6b-qbgnt" podUID="510ad10d-0554-47c8-9b55-deb359795071" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Oct 30 22:22:24.927120 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:24.922341    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-421583/distinct-w2-421583-kserve-69cf5f4d6b-qbgnt"
Oct 30 22:22:24.951117 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:24.943771    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/distinct-w3-d52c59-kserve-ffb5f5b96-l58xl" podUID="c7b1483b-2631-4aa2-a888-3096761d5494" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:24.951376 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:24.951354    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d52c59/distinct-w3-d52c59-kserve-ffb5f5b96-l58xl"
Oct 30 22:22:24.981065 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:24.955400    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/distinct-2-w6-012ba0-kserve-78dbcd8c46-hqdfm" podUID="8326b014-989b-4722-befb-6fa8ea60688b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Oct 30 22:22:24.981065 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:24.955759    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-012ba0/distinct-2-w6-012ba0-kserve-78dbcd8c46-hqdfm"
Oct 30 22:22:25.025240 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:25.025190    4477 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-2b1e88/simulator-w1-2b1e88-kserve-6c579b99f8-q5wpq" podUID="722f10d8-b7a2-4bf7-a274-9669529ef427" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:25.103180 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:25.103133    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/distinct-2-w0-09d0d5-kserve-7dcf89b98b-7cjph" podUID="8cbd04b4-bb3e-4bbb-a41a-46131c9de051" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:25.108062 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:25.104630    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/premium-w5-54114e-kserve-57f6747bcc-7wjrw" podUID="61990d22-ab8f-45bc-a4d6-4c8d528742f1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:25.151514 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:25.151475    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-09d0d5/distinct-2-w0-09d0d5-kserve-7dcf89b98b-7cjph"
Oct 30 22:22:25.179258 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:25.179233    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-54114e/premium-w5-54114e-kserve-57f6747bcc-7wjrw"
Oct 30 22:22:25.474297 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:25.461006    4477 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-09d0d5/distinct-2-w0-09d0d5-kserve-7dcf89b98b-7cjph" podUID="8cbd04b4-bb3e-4bbb-a41a-46131c9de051" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:25.768376 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:25.768335    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/premium-w6-012ba0-kserve-658654dbdb-d2k27" podUID="085847e5-4f50-419e-984f-b612b104ed90" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": context deadline exceeded"
Oct 30 22:22:25.803990 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:25.794297    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/simulator-w1-2b1e88-kserve-6c579b99f8-q5wpq" podUID="722f10d8-b7a2-4bf7-a274-9669529ef427" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded"
Oct 30 22:22:25.803990 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:25.796238    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-012ba0/premium-w6-012ba0-kserve-658654dbdb-d2k27"
Oct 30 22:22:25.861162 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:25.861114    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0ad7c5/premium-w4-0ad7c5-kserve-7bbf78cd4-pm6wc" podUID="5c10371e-56e7-42ec-9ef8-0802e15cb2ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:25.889810 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:25.889530    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0ad7c5/premium-w4-0ad7c5-kserve-7bbf78cd4-pm6wc"
Oct 30 22:22:25.999068 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:25.998661    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/distinct-w2-421583-kserve-69cf5f4d6b-qbgnt" podUID="510ad10d-0554-47c8-9b55-deb359795071" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Oct 30 22:22:26.046411 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:26.032504    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/distinct-2-w6-012ba0-kserve-78dbcd8c46-hqdfm" podUID="8326b014-989b-4722-befb-6fa8ea60688b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Oct 30 22:22:26.046411 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:26.032827    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/distinct-w3-d52c59-kserve-ffb5f5b96-l58xl" podUID="c7b1483b-2631-4aa2-a888-3096761d5494" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:26.078146 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:26.075855    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-012ba0/premium-w6-012ba0-kserve-658654dbdb-d2k27"
Oct 30 22:22:26.108670 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:26.107437    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/distinct-2-w0-09d0d5-kserve-7dcf89b98b-7cjph" podUID="8cbd04b4-bb3e-4bbb-a41a-46131c9de051" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:26.126385 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:22:26.126354    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f5f8741_f496_46a2_b357_72c0e27ea2ae.slice/crio-c4a23843c1f8e846b7eb82a155d52d5f50bd71c0e69d257d0ec24e2c9d1f16b7 WatchSource:0}: Error finding container c4a23843c1f8e846b7eb82a155d52d5f50bd71c0e69d257d0ec24e2c9d1f16b7: Status 404 returned error can't find the container with id c4a23843c1f8e846b7eb82a155d52d5f50bd71c0e69d257d0ec24e2c9d1f16b7
Oct 30 22:22:26.154921 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:26.154882    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/premium-w5-54114e-kserve-57f6747bcc-7wjrw" podUID="61990d22-ab8f-45bc-a4d6-4c8d528742f1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded"
Oct 30 22:22:26.299687 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:26.299645    4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:22:26.299919 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:26.299895    4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:22:26.300083 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:26.300063    4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:22:26.445480 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:26.445443    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-crp67" event={"ID":"2f5f8741-f496-46a2-b357-72c0e27ea2ae","Type":"ContainerStarted","Data":"c4a23843c1f8e846b7eb82a155d52d5f50bd71c0e69d257d0ec24e2c9d1f16b7"}
Oct 30 22:22:26.467015 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:26.465686    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-09d0d5/distinct-2-w0-09d0d5-kserve-7dcf89b98b-7cjph"
Oct 30 22:22:26.694139 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:26.694102    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-crp67"]
Oct 30 22:22:26.778672 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:22:26.778637    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 22:22:26.857067 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:26.856253    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/simulator-w1-2b1e88-kserve-6c579b99f8-q5wpq" podUID="722f10d8-b7a2-4bf7-a274-9669529ef427" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:26.940123 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:26.921327    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0ad7c5/premium-w4-0ad7c5-kserve-7bbf78cd4-pm6wc" podUID="5c10371e-56e7-42ec-9ef8-0802e15cb2ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded"
Oct 30 22:22:27.002089 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:27.001234    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/distinct-w2-421583-kserve-69cf5f4d6b-qbgnt" podUID="510ad10d-0554-47c8-9b55-deb359795071" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:27.034208 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:27.032907    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/distinct-2-w6-012ba0-kserve-78dbcd8c46-hqdfm" podUID="8326b014-989b-4722-befb-6fa8ea60688b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:27.115067 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:27.073811    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/premium-w6-012ba0-kserve-658654dbdb-d2k27" podUID="085847e5-4f50-419e-984f-b612b104ed90" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:27.115067 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:27.109226    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/distinct-w3-d52c59-kserve-ffb5f5b96-l58xl" podUID="c7b1483b-2631-4aa2-a888-3096761d5494" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:27.292066 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:27.230111    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/premium-w5-54114e-kserve-57f6747bcc-7wjrw" podUID="61990d22-ab8f-45bc-a4d6-4c8d528742f1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded"
Oct 30 22:22:27.695070 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:27.687877    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-2b1e88/simulator-w1-2b1e88-kserve-6c579b99f8-q5wpq"
Oct 30 22:22:27.945797 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:27.940617    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0ad7c5/premium-w4-0ad7c5-kserve-7bbf78cd4-pm6wc" podUID="5c10371e-56e7-42ec-9ef8-0802e15cb2ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:28.016706 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:28.016641    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-crp67" event={"ID":"2f5f8741-f496-46a2-b357-72c0e27ea2ae","Type":"ContainerStarted","Data":"7668f95924f257781ec7ea17f93cba6a28d30ae93598757f6025ae58d55951af"}
Oct 30 22:22:28.102683 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:28.101133    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/premium-w6-012ba0-kserve-658654dbdb-d2k27" podUID="085847e5-4f50-419e-984f-b612b104ed90" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:28.106605 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:28.103107    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-012ba0/premium-w6-012ba0-kserve-658654dbdb-d2k27"
Oct 30 22:22:28.106605 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:28.104122    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/distinct-2-w6-012ba0-kserve-78dbcd8c46-hqdfm" podUID="8326b014-989b-4722-befb-6fa8ea60688b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:28.124803 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:28.111893    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/distinct-w3-d52c59-kserve-ffb5f5b96-l58xl" podUID="c7b1483b-2631-4aa2-a888-3096761d5494" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Oct 30 22:22:28.124803 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:28.117180    4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-crp67"
Oct 30 22:22:28.125162 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:28.125137    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/distinct-w2-421583-kserve-69cf5f4d6b-qbgnt" podUID="510ad10d-0554-47c8-9b55-deb359795071" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:28.145624 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:28.145599    4477 patch_prober.go:29] interesting pod/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-crp67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Oct 30 22:22:28.145803 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:28.145778    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-crp67" podUID="2f5f8741-f496-46a2-b357-72c0e27ea2ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Oct 30 22:22:28.175317 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:28.175283    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-012ba0/distinct-2-w6-012ba0-kserve-78dbcd8c46-hqdfm"
Oct 30 22:22:28.189279 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:28.189223    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-crp67" podStartSLOduration=5.189202761 podStartE2EDuration="5.189202761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:22:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:22:28.10101289 +0000 UTC m=+932.026939122" watchObservedRunningTime="2026-10-30 22:22:28.189202761 +0000 UTC m=+932.115128992"
Oct 30 22:22:28.311023 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:28.281781    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-54114e/premium-w5-54114e-kserve-57f6747bcc-7wjrw"
Oct 30 22:22:28.311023 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:28.282325    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d52c59/distinct-w3-d52c59-kserve-ffb5f5b96-l58xl"
Oct 30 22:22:28.311023 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:28.282379    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-012ba0/premium-w6-012ba0-kserve-658654dbdb-d2k27"
Oct 30 22:22:28.328437 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:28.328268    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-421583/distinct-w2-421583-kserve-69cf5f4d6b-qbgnt"
Oct 30 22:22:28.371095 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:28.369690    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0ad7c5/premium-w4-0ad7c5-kserve-7bbf78cd4-pm6wc"
Oct 30 22:22:29.112467 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:29.112429    4477 patch_prober.go:29] interesting pod/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-crp67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Oct 30 22:22:29.112990 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:29.112954    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-crp67" podUID="2f5f8741-f496-46a2-b357-72c0e27ea2ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Oct 30 22:22:30.112023 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:30.111993    4477 patch_prober.go:29] interesting pod/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-crp67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Oct 30 22:22:30.112204 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:30.112050    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-crp67" podUID="2f5f8741-f496-46a2-b357-72c0e27ea2ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Oct 30 22:22:31.112655 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:31.112618    4477 patch_prober.go:29] interesting pod/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-crp67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Oct 30 22:22:31.113085 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:31.112676    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-crp67" podUID="2f5f8741-f496-46a2-b357-72c0e27ea2ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Oct 30 22:22:32.112096 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:32.112030    4477 patch_prober.go:29] interesting pod/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-crp67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Oct 30 22:22:32.112274 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:32.112125    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-crp67" podUID="2f5f8741-f496-46a2-b357-72c0e27ea2ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Oct 30 22:22:32.997883 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:32.997845    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74c3a2d4-fqxxl"]
Oct 30 22:22:33.003380 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:33.003337    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74c3a2d4-fqxxl"
Oct 30 22:22:33.005645 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:33.005617    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74c3a2d4-fqxxl"]
Oct 30 22:22:33.027655 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:33.027616    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:22:33.027787 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:33.027691    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 22:22:33.099937 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:33.099902    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25689f97-ea82-4662-976a-a4e8f7f2282b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74c3a2d4-fqxxl\" (UID: \"25689f97-ea82-4662-976a-a4e8f7f2282b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74c3a2d4-fqxxl"
Oct 30 22:22:33.099937 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:33.099939    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25689f97-ea82-4662-976a-a4e8f7f2282b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74c3a2d4-fqxxl\" (UID: \"25689f97-ea82-4662-976a-a4e8f7f2282b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74c3a2d4-fqxxl"
Oct 30 22:22:33.100176 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:33.100153    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgcwr\" (UniqueName: \"kubernetes.io/projected/25689f97-ea82-4662-976a-a4e8f7f2282b-kube-api-access-cgcwr\") pod \"maas-api-revoke-sub-74c3a2d4-fqxxl\" (UID: \"25689f97-ea82-4662-976a-a4e8f7f2282b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74c3a2d4-fqxxl"
Oct 30 22:22:33.112867 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:33.112836    4477 patch_prober.go:29] interesting pod/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-crp67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Oct 30 22:22:33.113012 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:33.112898    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-crp67" podUID="2f5f8741-f496-46a2-b357-72c0e27ea2ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Oct 30 22:22:33.201088 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:33.201027    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25689f97-ea82-4662-976a-a4e8f7f2282b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74c3a2d4-fqxxl\" (UID: \"25689f97-ea82-4662-976a-a4e8f7f2282b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74c3a2d4-fqxxl"
Oct 30 22:22:33.201264 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:33.201101    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25689f97-ea82-4662-976a-a4e8f7f2282b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74c3a2d4-fqxxl\" (UID: \"25689f97-ea82-4662-976a-a4e8f7f2282b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74c3a2d4-fqxxl"
Oct 30 22:22:33.201264 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:33.201192    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgcwr\" (UniqueName: \"kubernetes.io/projected/25689f97-ea82-4662-976a-a4e8f7f2282b-kube-api-access-cgcwr\") pod \"maas-api-revoke-sub-74c3a2d4-fqxxl\" (UID: \"25689f97-ea82-4662-976a-a4e8f7f2282b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74c3a2d4-fqxxl"
Oct 30 22:22:33.201785 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:33.201759    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25689f97-ea82-4662-976a-a4e8f7f2282b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74c3a2d4-fqxxl\" (UID: \"25689f97-ea82-4662-976a-a4e8f7f2282b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74c3a2d4-fqxxl"
Oct 30 22:22:33.203408 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:33.203387    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25689f97-ea82-4662-976a-a4e8f7f2282b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74c3a2d4-fqxxl\" (UID: \"25689f97-ea82-4662-976a-a4e8f7f2282b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74c3a2d4-fqxxl"
Oct 30 22:22:33.219235 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:33.219211    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgcwr\" (UniqueName: \"kubernetes.io/projected/25689f97-ea82-4662-976a-a4e8f7f2282b-kube-api-access-cgcwr\") pod \"maas-api-revoke-sub-74c3a2d4-fqxxl\" (UID: \"25689f97-ea82-4662-976a-a4e8f7f2282b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74c3a2d4-fqxxl"
Oct 30 22:22:33.323789 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:33.323716    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74c3a2d4-fqxxl"
Oct 30 22:22:33.644469 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:33.644440    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74c3a2d4-fqxxl"]
Oct 30 22:22:33.647790 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:22:33.647764    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25689f97_ea82_4662_976a_a4e8f7f2282b.slice/crio-4be4d11503b590d54c3109b859cc6ae5767df618fb22cf2c644eb1061afc30b7 WatchSource:0}: Error finding container 4be4d11503b590d54c3109b859cc6ae5767df618fb22cf2c644eb1061afc30b7: Status 404 returned error can't find the container with id 4be4d11503b590d54c3109b859cc6ae5767df618fb22cf2c644eb1061afc30b7
Oct 30 22:22:34.062339 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:34.062305    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74c3a2d4-fqxxl" event={"ID":"25689f97-ea82-4662-976a-a4e8f7f2282b","Type":"ContainerStarted","Data":"4be4d11503b590d54c3109b859cc6ae5767df618fb22cf2c644eb1061afc30b7"}
Oct 30 22:22:34.113055 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:34.112995    4477 patch_prober.go:29] interesting pod/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-crp67 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Oct 30 22:22:34.113212 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:34.113062    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-crp67" podUID="2f5f8741-f496-46a2-b357-72c0e27ea2ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Oct 30 22:22:35.072437 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:35.072354    4477 generic.go:356] "Generic (PLEG): container finished" podID="25689f97-ea82-4662-976a-a4e8f7f2282b" containerID="6287fed66e05f45cc47318eff344fbfb86b5674ddfd1a2a2cf5f3875d8c4f7b9" exitCode=0
Oct 30 22:22:35.072437 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:35.072421    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74c3a2d4-fqxxl" event={"ID":"25689f97-ea82-4662-976a-a4e8f7f2282b","Type":"ContainerDied","Data":"6287fed66e05f45cc47318eff344fbfb86b5674ddfd1a2a2cf5f3875d8c4f7b9"}
Oct 30 22:22:35.345704 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:35.345627    4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-crp67"
Oct 30 22:22:35.345987 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:35.345942    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-crp67"
Oct 30 22:22:35.347161 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:35.347137    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-crp67"
Oct 30 22:22:36.240307 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:36.240280    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74c3a2d4-fqxxl"
Oct 30 22:22:36.342620 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:36.342556    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25689f97-ea82-4662-976a-a4e8f7f2282b-kube-api-access-cgcwr\" (UniqueName: \"kubernetes.io/projected/25689f97-ea82-4662-976a-a4e8f7f2282b-kube-api-access-cgcwr\") pod \"25689f97-ea82-4662-976a-a4e8f7f2282b\" (UID: \"25689f97-ea82-4662-976a-a4e8f7f2282b\") "
Oct 30 22:22:36.342777 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:36.342679    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25689f97-ea82-4662-976a-a4e8f7f2282b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25689f97-ea82-4662-976a-a4e8f7f2282b-maas-api-xks-ca\") pod \"25689f97-ea82-4662-976a-a4e8f7f2282b\" (UID: \"25689f97-ea82-4662-976a-a4e8f7f2282b\") "
Oct 30 22:22:36.342878 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:36.342859    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25689f97-ea82-4662-976a-a4e8f7f2282b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25689f97-ea82-4662-976a-a4e8f7f2282b-maas-api-openshift-ca\") pod \"25689f97-ea82-4662-976a-a4e8f7f2282b\" (UID: \"25689f97-ea82-4662-976a-a4e8f7f2282b\") "
Oct 30 22:22:36.343228 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:36.343205    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25689f97-ea82-4662-976a-a4e8f7f2282b-maas-api-openshift-ca" pod "25689f97-ea82-4662-976a-a4e8f7f2282b" (UID: "25689f97-ea82-4662-976a-a4e8f7f2282b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:22:36.343586 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:36.343565    4477 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25689f97-ea82-4662-976a-a4e8f7f2282b-maas-api-openshift-ca\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:22:36.344631 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:36.344609    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25689f97-ea82-4662-976a-a4e8f7f2282b-kube-api-access-cgcwr" pod "25689f97-ea82-4662-976a-a4e8f7f2282b" (UID: "25689f97-ea82-4662-976a-a4e8f7f2282b"). InnerVolumeSpecName "kube-api-access-cgcwr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:22:36.344721 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:36.344682    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25689f97-ea82-4662-976a-a4e8f7f2282b-maas-api-xks-ca" pod "25689f97-ea82-4662-976a-a4e8f7f2282b" (UID: "25689f97-ea82-4662-976a-a4e8f7f2282b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:22:36.444624 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:36.444598    4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgcwr\" (UniqueName: \"kubernetes.io/projected/25689f97-ea82-4662-976a-a4e8f7f2282b-kube-api-access-cgcwr\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:22:36.444624 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:36.444621    4477 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25689f97-ea82-4662-976a-a4e8f7f2282b-maas-api-xks-ca\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:22:37.086741 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:37.086707    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74c3a2d4-fqxxl"
Oct 30 22:22:37.086741 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:37.086722    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74c3a2d4-fqxxl" event={"ID":"25689f97-ea82-4662-976a-a4e8f7f2282b","Type":"ContainerDied","Data":"4be4d11503b590d54c3109b859cc6ae5767df618fb22cf2c644eb1061afc30b7"}
Oct 30 22:22:37.086955 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:37.086775    4477 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4be4d11503b590d54c3109b859cc6ae5767df618fb22cf2c644eb1061afc30b7"
Oct 30 22:22:37.937120 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:37.937092    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-p4vsn"]
Oct 30 22:22:37.937690 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:37.937675    4477 cpu_manager.go:414] "containerMap: removing container" podUID="25689f97-ea82-4662-976a-a4e8f7f2282b" containerName="revoke-keys"
Oct 30 22:22:37.937749 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:37.937692    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25689f97-ea82-4662-976a-a4e8f7f2282b" containerName="revoke-keys"
Oct 30 22:22:37.937849 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:37.937839    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="25689f97-ea82-4662-976a-a4e8f7f2282b" containerName="revoke-keys"
Oct 30 22:22:37.941692 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:37.941664    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-p4vsn"
Oct 30 22:22:37.965299 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:37.965274    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-p4vsn"]
Oct 30 22:22:38.062870 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:38.062840    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7237cb56-76fc-4438-b031-4c0eee080b7d-credential-socket\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-p4vsn\" (UID: \"7237cb56-76fc-4438-b031-4c0eee080b7d\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-p4vsn"
Oct 30 22:22:38.063089 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:38.062894    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7237cb56-76fc-4438-b031-4c0eee080b7d-istio-token\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-p4vsn\" (UID: \"7237cb56-76fc-4438-b031-4c0eee080b7d\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-p4vsn"
Oct 30 22:22:38.063089 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:38.062976    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7237cb56-76fc-4438-b031-4c0eee080b7d-istio-data\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-p4vsn\" (UID: \"7237cb56-76fc-4438-b031-4c0eee080b7d\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-p4vsn"
Oct 30 22:22:38.063185 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:38.063111    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7237cb56-76fc-4438-b031-4c0eee080b7d-istio-podinfo\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-p4vsn\" (UID: \"7237cb56-76fc-4438-b031-4c0eee080b7d\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-p4vsn"
Oct 30 22:22:38.063185 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:38.063149    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7237cb56-76fc-4438-b031-4c0eee080b7d-istiod-ca-cert\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-p4vsn\" (UID: \"7237cb56-76fc-4438-b031-4c0eee080b7d\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-p4vsn"
Oct 30 22:22:38.063185 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:38.063174    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7237cb56-76fc-4438-b031-4c0eee080b7d-workload-socket\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-p4vsn\" (UID: \"7237cb56-76fc-4438-b031-4c0eee080b7d\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-p4vsn"
Oct 30 22:22:38.063367 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:38.063351    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7237cb56-76fc-4438-b031-4c0eee080b7d-workload-certs\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-p4vsn\" (UID: \"7237cb56-76fc-4438-b031-4c0eee080b7d\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-p4vsn"
Oct 30 22:22:38.063420 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:38.063382    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7237cb56-76fc-4438-b031-4c0eee080b7d-istio-envoy\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-p4vsn\" (UID: \"7237cb56-76fc-4438-b031-4c0eee080b7d\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-p4vsn"
Oct 30 22:22:38.063420 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:38.063412    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25gjl\" (UniqueName: \"kubernetes.io/projected/7237cb56-76fc-4438-b031-4c0eee080b7d-kube-api-access-25gjl\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-p4vsn\" (UID: \"7237cb56-76fc-4438-b031-4c0eee080b7d\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-p4vsn"
Oct 30 22:22:38.112543 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:38.112309    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-rpk4x"]
Oct 30 22:22:38.118988 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:38.118959    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-rpk4x"
Oct 30 22:22:38.135256 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:38.135229    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-rpk4x"]
Oct 30 22:22:38.165815 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:38.165166    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7237cb56-76fc-4438-b031-4c0eee080b7d-istio-podinfo\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-p4vsn\" (UID: \"7237cb56-76fc-4438-b031-4c0eee080b7d\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-p4vsn"
Oct 30 22:22:38.165815 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:38.165228    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7237cb56-76fc-4438-b031-4c0eee080b7d-istiod-ca-cert\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-p4vsn\" (UID: \"7237cb56-76fc-4438-b031-4c0eee080b7d\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-p4vsn"
Oct 30 22:22:38.165815 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:38.165271    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7237cb56-76fc-4438-b031-4c0eee080b7d-workload-socket\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-p4vsn\" (UID: \"7237cb56-76fc-4438-b031-4c0eee080b7d\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-p4vsn"
Oct 30 22:22:38.165815 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:38.165530    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7237cb56-76fc-4438-b031-4c0eee080b7d-workload-certs\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-p4vsn\" (UID: \"7237cb56-76fc-4438-b031-4c0eee080b7d\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-p4vsn"
Oct 30 22:22:38.165815 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:38.165570    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7237cb56-76fc-4438-b031-4c0eee080b7d-istio-envoy\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-p4vsn\" (UID: \"7237cb56-76fc-4438-b031-4c0eee080b7d\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-p4vsn"
Oct 30 22:22:38.165815 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:38.165630    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25gjl\" (UniqueName: \"kubernetes.io/projected/7237cb56-76fc-4438-b031-4c0eee080b7d-kube-api-access-25gjl\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-p4vsn\" (UID: \"7237cb56-76fc-4438-b031-4c0eee080b7d\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-p4vsn"
Oct 30 22:22:38.165815 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:38.165706    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7237cb56-76fc-4438-b031-4c0eee080b7d-credential-socket\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-p4vsn\" (UID: \"7237cb56-76fc-4438-b031-4c0eee080b7d\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-p4vsn"
Oct 30 22:22:38.165815 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:38.165781    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7237cb56-76fc-4438-b031-4c0eee080b7d-istio-token\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-p4vsn\" (UID: \"7237cb56-76fc-4438-b031-4c0eee080b7d\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-p4vsn"
Oct 30 22:22:38.166402 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:38.165831    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7237cb56-76fc-4438-b031-4c0eee080b7d-istio-data\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-p4vsn\" (UID: \"7237cb56-76fc-4438-b031-4c0eee080b7d\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-p4vsn"
Oct 30 22:22:38.166402 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:38.166301    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7237cb56-76fc-4438-b031-4c0eee080b7d-istio-data\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-p4vsn\" (UID: \"7237cb56-76fc-4438-b031-4c0eee080b7d\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-p4vsn"
Oct 30 22:22:38.166402 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:38.166355    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7237cb56-76fc-4438-b031-4c0eee080b7d-istiod-ca-cert\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-p4vsn\" (UID: \"7237cb56-76fc-4438-b031-4c0eee080b7d\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-p4vsn"
Oct 30 22:22:38.166579 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:38.166552    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7237cb56-76fc-4438-b031-4c0eee080b7d-workload-socket\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-p4vsn\" (UID: \"7237cb56-76fc-4438-b031-4c0eee080b7d\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-p4vsn"
Oct 30 22:22:38.167319 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:38.166839    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7237cb56-76fc-4438-b031-4c0eee080b7d-credential-socket\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-p4vsn\" (UID: \"7237cb56-76fc-4438-b031-4c0eee080b7d\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-p4vsn"
Oct 30 22:22:38.167319 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:38.166974    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7237cb56-76fc-4438-b031-4c0eee080b7d-workload-certs\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-p4vsn\" (UID: \"7237cb56-76fc-4438-b031-4c0eee080b7d\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-p4vsn"
Oct 30 22:22:38.170868 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:38.170845    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7237cb56-76fc-4438-b031-4c0eee080b7d-istio-envoy\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-p4vsn\" (UID: \"7237cb56-76fc-4438-b031-4c0eee080b7d\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-p4vsn"
Oct 30 22:22:38.171502 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:38.171477    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7237cb56-76fc-4438-b031-4c0eee080b7d-istio-podinfo\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-p4vsn\" (UID: \"7237cb56-76fc-4438-b031-4c0eee080b7d\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-p4vsn"
Oct 30 22:22:38.184014 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:38.183991    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25gjl\" (UniqueName: \"kubernetes.io/projected/7237cb56-76fc-4438-b031-4c0eee080b7d-kube-api-access-25gjl\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-p4vsn\" (UID: \"7237cb56-76fc-4438-b031-4c0eee080b7d\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-p4vsn"
Oct 30 22:22:38.184100 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:38.184030    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7237cb56-76fc-4438-b031-4c0eee080b7d-istio-token\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-p4vsn\" (UID: \"7237cb56-76fc-4438-b031-4c0eee080b7d\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-p4vsn"
Oct 30 22:22:38.258162 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:38.258099    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-p4vsn"
Oct 30 22:22:38.266583 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:38.266557    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69b42852-cbc4-46b7-be1b-98ce37efa0d4-istio-token\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-rpk4x\" (UID: \"69b42852-cbc4-46b7-be1b-98ce37efa0d4\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-rpk4x"
Oct 30 22:22:38.266679 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:38.266617    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69b42852-cbc4-46b7-be1b-98ce37efa0d4-workload-socket\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-rpk4x\" (UID: \"69b42852-cbc4-46b7-be1b-98ce37efa0d4\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-rpk4x"
Oct 30 22:22:38.266750 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:38.266729    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69b42852-cbc4-46b7-be1b-98ce37efa0d4-credential-socket\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-rpk4x\" (UID: \"69b42852-cbc4-46b7-be1b-98ce37efa0d4\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-rpk4x"
Oct 30 22:22:38.266905 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:38.266883    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69b42852-cbc4-46b7-be1b-98ce37efa0d4-workload-certs\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-rpk4x\" (UID: \"69b42852-cbc4-46b7-be1b-98ce37efa0d4\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-rpk4x"
Oct 30 22:22:38.266957 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:38.266939    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kvcz\" (UniqueName: \"kubernetes.io/projected/69b42852-cbc4-46b7-be1b-98ce37efa0d4-kube-api-access-6kvcz\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-rpk4x\" (UID: \"69b42852-cbc4-46b7-be1b-98ce37efa0d4\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-rpk4x"
Oct 30 22:22:38.267033 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:38.267020    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69b42852-cbc4-46b7-be1b-98ce37efa0d4-istio-data\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-rpk4x\" (UID: \"69b42852-cbc4-46b7-be1b-98ce37efa0d4\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-rpk4x"
Oct 30 22:22:38.267118 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:38.267086    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69b42852-cbc4-46b7-be1b-98ce37efa0d4-istio-podinfo\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-rpk4x\" (UID: \"69b42852-cbc4-46b7-be1b-98ce37efa0d4\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-rpk4x"
Oct 30 22:22:38.267118 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:38.267110    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69b42852-cbc4-46b7-be1b-98ce37efa0d4-istio-envoy\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-rpk4x\" (UID: \"69b42852-cbc4-46b7-be1b-98ce37efa0d4\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-rpk4x"
Oct 30 22:22:38.267214 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:38.267137    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69b42852-cbc4-46b7-be1b-98ce37efa0d4-istiod-ca-cert\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-rpk4x\" (UID: \"69b42852-cbc4-46b7-be1b-98ce37efa0d4\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-rpk4x"
Oct 30 22:22:38.369419 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:38.369382    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69b42852-cbc4-46b7-be1b-98ce37efa0d4-workload-certs\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-rpk4x\" (UID: \"69b42852-cbc4-46b7-be1b-98ce37efa0d4\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-rpk4x"
Oct 30 22:22:38.369598 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:38.369457    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kvcz\" (UniqueName: \"kubernetes.io/projected/69b42852-cbc4-46b7-be1b-98ce37efa0d4-kube-api-access-6kvcz\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-rpk4x\" (UID: \"69b42852-cbc4-46b7-be1b-98ce37efa0d4\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-rpk4x"
Oct 30 22:22:38.369598 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:38.369505    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69b42852-cbc4-46b7-be1b-98ce37efa0d4-istio-data\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-rpk4x\" (UID: \"69b42852-cbc4-46b7-be1b-98ce37efa0d4\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-rpk4x"
Oct 30 22:22:38.369598 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:38.369555    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69b42852-cbc4-46b7-be1b-98ce37efa0d4-istio-podinfo\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-rpk4x\" (UID: \"69b42852-cbc4-46b7-be1b-98ce37efa0d4\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-rpk4x"
Oct 30 22:22:38.369757 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:38.369595    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69b42852-cbc4-46b7-be1b-98ce37efa0d4-istio-envoy\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-rpk4x\" (UID: \"69b42852-cbc4-46b7-be1b-98ce37efa0d4\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-rpk4x"
Oct 30 22:22:38.369757 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:38.369647    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69b42852-cbc4-46b7-be1b-98ce37efa0d4-istiod-ca-cert\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-rpk4x\" (UID: \"69b42852-cbc4-46b7-be1b-98ce37efa0d4\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-rpk4x"
Oct 30 22:22:38.369861 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:38.369766    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69b42852-cbc4-46b7-be1b-98ce37efa0d4-istio-token\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-rpk4x\" (UID: \"69b42852-cbc4-46b7-be1b-98ce37efa0d4\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-rpk4x"
Oct 30 22:22:38.369861 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:38.369822    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69b42852-cbc4-46b7-be1b-98ce37efa0d4-workload-certs\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-rpk4x\" (UID: \"69b42852-cbc4-46b7-be1b-98ce37efa0d4\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-rpk4x"
Oct 30 22:22:38.369861 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:38.369852    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69b42852-cbc4-46b7-be1b-98ce37efa0d4-istio-data\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-rpk4x\" (UID: \"69b42852-cbc4-46b7-be1b-98ce37efa0d4\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-rpk4x"
Oct 30 22:22:38.370016 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:38.369863    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69b42852-cbc4-46b7-be1b-98ce37efa0d4-workload-socket\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-rpk4x\" (UID: \"69b42852-cbc4-46b7-be1b-98ce37efa0d4\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-rpk4x"
Oct 30 22:22:38.370016 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:38.369908    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69b42852-cbc4-46b7-be1b-98ce37efa0d4-credential-socket\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-rpk4x\" (UID: \"69b42852-cbc4-46b7-be1b-98ce37efa0d4\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-rpk4x"
Oct 30 22:22:38.370304 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:38.370278    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69b42852-cbc4-46b7-be1b-98ce37efa0d4-credential-socket\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-rpk4x\" (UID: \"69b42852-cbc4-46b7-be1b-98ce37efa0d4\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-rpk4x"
Oct 30 22:22:38.370533 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:38.370509    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69b42852-cbc4-46b7-be1b-98ce37efa0d4-istiod-ca-cert\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-rpk4x\" (UID: \"69b42852-cbc4-46b7-be1b-98ce37efa0d4\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-rpk4x"
Oct 30 22:22:38.370650 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:38.370548    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69b42852-cbc4-46b7-be1b-98ce37efa0d4-workload-socket\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-rpk4x\" (UID: \"69b42852-cbc4-46b7-be1b-98ce37efa0d4\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-rpk4x"
Oct 30 22:22:38.372801 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:38.372782    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69b42852-cbc4-46b7-be1b-98ce37efa0d4-istio-podinfo\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-rpk4x\" (UID: \"69b42852-cbc4-46b7-be1b-98ce37efa0d4\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-rpk4x"
Oct 30 22:22:38.372937 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:38.372922    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69b42852-cbc4-46b7-be1b-98ce37efa0d4-istio-envoy\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-rpk4x\" (UID: \"69b42852-cbc4-46b7-be1b-98ce37efa0d4\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-rpk4x"
Oct 30 22:22:38.387033 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:38.387008    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69b42852-cbc4-46b7-be1b-98ce37efa0d4-istio-token\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-rpk4x\" (UID: \"69b42852-cbc4-46b7-be1b-98ce37efa0d4\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-rpk4x"
Oct 30 22:22:38.387642 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:38.387620    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kvcz\" (UniqueName: \"kubernetes.io/projected/69b42852-cbc4-46b7-be1b-98ce37efa0d4-kube-api-access-6kvcz\") pod \"e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-rpk4x\" (UID: \"69b42852-cbc4-46b7-be1b-98ce37efa0d4\") " pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-rpk4x"
Oct 30 22:22:38.435464 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:38.435438    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-rpk4x"
Oct 30 22:22:38.585752 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:38.585518    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-p4vsn"]
Oct 30 22:22:38.587274 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:22:38.587236    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7237cb56_76fc_4438_b031_4c0eee080b7d.slice/crio-4a5781e719430479588192547253179cc56118f31525b62c048d926941dc288c WatchSource:0}: Error finding container 4a5781e719430479588192547253179cc56118f31525b62c048d926941dc288c: Status 404 returned error can't find the container with id 4a5781e719430479588192547253179cc56118f31525b62c048d926941dc288c
Oct 30 22:22:38.590576 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:38.590528    4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:22:38.590666 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:38.590622    4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:22:38.590747 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:38.590672    4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:22:38.743009 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:38.742977    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-rpk4x"]
Oct 30 22:22:38.746404 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:22:38.746377    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69b42852_cbc4_46b7_be1b_98ce37efa0d4.slice/crio-3645df837bb85026d887dc8af7079c5233c325eb490b5de0187f0d3719d445f7 WatchSource:0}: Error finding container 3645df837bb85026d887dc8af7079c5233c325eb490b5de0187f0d3719d445f7: Status 404 returned error can't find the container with id 3645df837bb85026d887dc8af7079c5233c325eb490b5de0187f0d3719d445f7
Oct 30 22:22:38.749146 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:38.749111    4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:22:38.749236 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:38.749180    4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:22:38.749236 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:38.749216    4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:22:39.106431 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:39.106394    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-p4vsn" event={"ID":"7237cb56-76fc-4438-b031-4c0eee080b7d","Type":"ContainerStarted","Data":"21b0ec2dee5bf78ed90b850ab88070399be834cbb6d18a596fb1972d905f9f81"}
Oct 30 22:22:39.106927 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:39.106457    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-p4vsn" event={"ID":"7237cb56-76fc-4438-b031-4c0eee080b7d","Type":"ContainerStarted","Data":"4a5781e719430479588192547253179cc56118f31525b62c048d926941dc288c"}
Oct 30 22:22:39.108545 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:39.108510    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-rpk4x" event={"ID":"69b42852-cbc4-46b7-be1b-98ce37efa0d4","Type":"ContainerStarted","Data":"bba4ec6d91fcb18b01f95250806d3518a03ba22e431e61012fed79f81201a618"}
Oct 30 22:22:39.108665 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:39.108561    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-rpk4x" event={"ID":"69b42852-cbc4-46b7-be1b-98ce37efa0d4","Type":"ContainerStarted","Data":"3645df837bb85026d887dc8af7079c5233c325eb490b5de0187f0d3719d445f7"}
Oct 30 22:22:39.149601 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:39.149547    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-p4vsn" podStartSLOduration=2.149534895 podStartE2EDuration="2.149534895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:22:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:22:39.14561061 +0000 UTC m=+943.071536816" watchObservedRunningTime="2026-10-30 22:22:39.149534895 +0000 UTC m=+943.075461101"
Oct 30 22:22:39.197067 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:39.196010    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-rpk4x" podStartSLOduration=1.195980962 podStartE2EDuration="1.195980962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:22:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:22:39.189001681 +0000 UTC m=+943.114927915" watchObservedRunningTime="2026-10-30 22:22:39.195980962 +0000 UTC m=+943.121907170"
Oct 30 22:22:39.259097 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:39.259036    4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-p4vsn"
Oct 30 22:22:39.260662 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:39.260634    4477 patch_prober.go:29] interesting pod/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-p4vsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Oct 30 22:22:39.260795 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:39.260682    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-p4vsn" podUID="7237cb56-76fc-4438-b031-4c0eee080b7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Oct 30 22:22:39.436535 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:39.436502    4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-rpk4x"
Oct 30 22:22:39.438489 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:39.438457    4477 patch_prober.go:29] interesting pod/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-rpk4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Oct 30 22:22:39.438625 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:39.438508    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-rpk4x" podUID="69b42852-cbc4-46b7-be1b-98ce37efa0d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Oct 30 22:22:40.258668 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:40.258636    4477 patch_prober.go:29] interesting pod/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-p4vsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Oct 30 22:22:40.259028 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:40.258690    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-p4vsn" podUID="7237cb56-76fc-4438-b031-4c0eee080b7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Oct 30 22:22:40.435952 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:40.435915    4477 patch_prober.go:29] interesting pod/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-rpk4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Oct 30 22:22:40.436136 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:40.435976    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-rpk4x" podUID="69b42852-cbc4-46b7-be1b-98ce37efa0d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Oct 30 22:22:41.258890 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:41.258854    4477 patch_prober.go:29] interesting pod/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-p4vsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Oct 30 22:22:41.259327 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:41.258903    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-p4vsn" podUID="7237cb56-76fc-4438-b031-4c0eee080b7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Oct 30 22:22:41.341595 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:41.341554    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cad8417b-6lnzr"]
Oct 30 22:22:41.346140 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:41.346118    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cad8417b-6lnzr"
Oct 30 22:22:41.388190 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:41.388158    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 22:22:41.388391 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:41.388378    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:22:41.424207 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:41.424110    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cad8417b-6lnzr"]
Oct 30 22:22:41.437159 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:41.436864    4477 patch_prober.go:29] interesting pod/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-rpk4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Oct 30 22:22:41.437159 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:41.436923    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-rpk4x" podUID="69b42852-cbc4-46b7-be1b-98ce37efa0d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Oct 30 22:22:41.453251 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:41.453226    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7w5lm\" (UniqueName: \"kubernetes.io/projected/7efed674-a0ee-438e-aa0c-664a46b49c6b-kube-api-access-7w5lm\") pod \"maas-api-revoke-sub-cad8417b-6lnzr\" (UID: \"7efed674-a0ee-438e-aa0c-664a46b49c6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cad8417b-6lnzr"
Oct 30 22:22:41.453364 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:41.453256    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7efed674-a0ee-438e-aa0c-664a46b49c6b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cad8417b-6lnzr\" (UID: \"7efed674-a0ee-438e-aa0c-664a46b49c6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cad8417b-6lnzr"
Oct 30 22:22:41.453364 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:41.453279    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7efed674-a0ee-438e-aa0c-664a46b49c6b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cad8417b-6lnzr\" (UID: \"7efed674-a0ee-438e-aa0c-664a46b49c6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cad8417b-6lnzr"
Oct 30 22:22:41.554289 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:41.554215    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w5lm\" (UniqueName: \"kubernetes.io/projected/7efed674-a0ee-438e-aa0c-664a46b49c6b-kube-api-access-7w5lm\") pod \"maas-api-revoke-sub-cad8417b-6lnzr\" (UID: \"7efed674-a0ee-438e-aa0c-664a46b49c6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cad8417b-6lnzr"
Oct 30 22:22:41.554289 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:41.554249    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7efed674-a0ee-438e-aa0c-664a46b49c6b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cad8417b-6lnzr\" (UID: \"7efed674-a0ee-438e-aa0c-664a46b49c6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cad8417b-6lnzr"
Oct 30 22:22:41.554289 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:41.554269    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7efed674-a0ee-438e-aa0c-664a46b49c6b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cad8417b-6lnzr\" (UID: \"7efed674-a0ee-438e-aa0c-664a46b49c6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cad8417b-6lnzr"
Oct 30 22:22:41.555082 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:41.555034    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7efed674-a0ee-438e-aa0c-664a46b49c6b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cad8417b-6lnzr\" (UID: \"7efed674-a0ee-438e-aa0c-664a46b49c6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cad8417b-6lnzr"
Oct 30 22:22:41.556679 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:41.556652    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7efed674-a0ee-438e-aa0c-664a46b49c6b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cad8417b-6lnzr\" (UID: \"7efed674-a0ee-438e-aa0c-664a46b49c6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cad8417b-6lnzr"
Oct 30 22:22:41.602730 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:41.602698    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7w5lm\" (UniqueName: \"kubernetes.io/projected/7efed674-a0ee-438e-aa0c-664a46b49c6b-kube-api-access-7w5lm\") pod \"maas-api-revoke-sub-cad8417b-6lnzr\" (UID: \"7efed674-a0ee-438e-aa0c-664a46b49c6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cad8417b-6lnzr"
Oct 30 22:22:41.660016 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:41.659985    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cad8417b-6lnzr"
Oct 30 22:22:42.025838 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:42.025812    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cad8417b-6lnzr"]
Oct 30 22:22:42.028537 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:22:42.028504    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7efed674_a0ee_438e_aa0c_664a46b49c6b.slice/crio-0c740d27f2e9e858ee689ff884ec380ce52a8b4b6e59c7771d33d9b674fd85f0 WatchSource:0}: Error finding container 0c740d27f2e9e858ee689ff884ec380ce52a8b4b6e59c7771d33d9b674fd85f0: Status 404 returned error can't find the container with id 0c740d27f2e9e858ee689ff884ec380ce52a8b4b6e59c7771d33d9b674fd85f0
Oct 30 22:22:42.147506 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:42.147459    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cad8417b-6lnzr" event={"ID":"7efed674-a0ee-438e-aa0c-664a46b49c6b","Type":"ContainerStarted","Data":"0c740d27f2e9e858ee689ff884ec380ce52a8b4b6e59c7771d33d9b674fd85f0"}
Oct 30 22:22:42.258644 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:42.258614    4477 patch_prober.go:29] interesting pod/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-p4vsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Oct 30 22:22:42.258786 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:42.258662    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-p4vsn" podUID="7237cb56-76fc-4438-b031-4c0eee080b7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Oct 30 22:22:42.436763 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:42.436733    4477 patch_prober.go:29] interesting pod/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-rpk4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Oct 30 22:22:42.437151 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:42.436786    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-rpk4x" podUID="69b42852-cbc4-46b7-be1b-98ce37efa0d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Oct 30 22:22:43.155328 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:43.155294    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cad8417b-6lnzr" event={"ID":"7efed674-a0ee-438e-aa0c-664a46b49c6b","Type":"ContainerStarted","Data":"ec41613219cad5a5a3fd5fea9cf2c651869b1e5e32296d652f02b0e3c291dcc9"}
Oct 30 22:22:43.225519 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:43.223774    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cad8417b-6lnzr" podStartSLOduration=2.223752548 podStartE2EDuration="2.223752548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:22:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:22:43.198219039 +0000 UTC m=+947.124145262" watchObservedRunningTime="2026-10-30 22:22:43.223752548 +0000 UTC m=+947.149678755"
Oct 30 22:22:43.259022 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:43.258998    4477 patch_prober.go:29] interesting pod/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-p4vsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Oct 30 22:22:43.259152 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:43.259082    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-p4vsn" podUID="7237cb56-76fc-4438-b031-4c0eee080b7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Oct 30 22:22:43.436849 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:43.436758    4477 patch_prober.go:29] interesting pod/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-rpk4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Oct 30 22:22:43.436849 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:43.436822    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-rpk4x" podUID="69b42852-cbc4-46b7-be1b-98ce37efa0d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Oct 30 22:22:44.166708 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:44.163412    4477 generic.go:356] "Generic (PLEG): container finished" podID="7efed674-a0ee-438e-aa0c-664a46b49c6b" containerID="ec41613219cad5a5a3fd5fea9cf2c651869b1e5e32296d652f02b0e3c291dcc9" exitCode=0
Oct 30 22:22:44.166708 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:44.163486    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cad8417b-6lnzr" event={"ID":"7efed674-a0ee-438e-aa0c-664a46b49c6b","Type":"ContainerDied","Data":"ec41613219cad5a5a3fd5fea9cf2c651869b1e5e32296d652f02b0e3c291dcc9"}
Oct 30 22:22:44.259229 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:44.259190    4477 patch_prober.go:29] interesting pod/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-p4vsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Oct 30 22:22:44.259393 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:44.259258    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-p4vsn" podUID="7237cb56-76fc-4438-b031-4c0eee080b7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Oct 30 22:22:44.435900 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:44.435816    4477 patch_prober.go:29] interesting pod/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-rpk4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Oct 30 22:22:44.436082 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:44.436026    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-rpk4x" podUID="69b42852-cbc4-46b7-be1b-98ce37efa0d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Oct 30 22:22:45.263464 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:45.260392    4477 patch_prober.go:29] interesting pod/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-p4vsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Oct 30 22:22:45.263464 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:45.260462    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-p4vsn" podUID="7237cb56-76fc-4438-b031-4c0eee080b7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Oct 30 22:22:45.368390 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:45.368365    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cad8417b-6lnzr"
Oct 30 22:22:45.436906 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:45.436819    4477 patch_prober.go:29] interesting pod/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-rpk4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Oct 30 22:22:45.436906 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:45.436877    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-rpk4x" podUID="69b42852-cbc4-46b7-be1b-98ce37efa0d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Oct 30 22:22:45.461923 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:45.461881    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7efed674-a0ee-438e-aa0c-664a46b49c6b-kube-api-access-7w5lm\" (UniqueName: \"kubernetes.io/projected/7efed674-a0ee-438e-aa0c-664a46b49c6b-kube-api-access-7w5lm\") pod \"7efed674-a0ee-438e-aa0c-664a46b49c6b\" (UID: \"7efed674-a0ee-438e-aa0c-664a46b49c6b\") "
Oct 30 22:22:45.462074 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:45.462034    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7efed674-a0ee-438e-aa0c-664a46b49c6b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7efed674-a0ee-438e-aa0c-664a46b49c6b-maas-api-xks-ca\") pod \"7efed674-a0ee-438e-aa0c-664a46b49c6b\" (UID: \"7efed674-a0ee-438e-aa0c-664a46b49c6b\") "
Oct 30 22:22:45.462121 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:45.462095    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7efed674-a0ee-438e-aa0c-664a46b49c6b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7efed674-a0ee-438e-aa0c-664a46b49c6b-maas-api-openshift-ca\") pod \"7efed674-a0ee-438e-aa0c-664a46b49c6b\" (UID: \"7efed674-a0ee-438e-aa0c-664a46b49c6b\") "
Oct 30 22:22:45.462995 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:45.462964    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7efed674-a0ee-438e-aa0c-664a46b49c6b-maas-api-openshift-ca" pod "7efed674-a0ee-438e-aa0c-664a46b49c6b" (UID: "7efed674-a0ee-438e-aa0c-664a46b49c6b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:22:45.464469 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:45.464446    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7efed674-a0ee-438e-aa0c-664a46b49c6b-maas-api-xks-ca" pod "7efed674-a0ee-438e-aa0c-664a46b49c6b" (UID: "7efed674-a0ee-438e-aa0c-664a46b49c6b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:22:45.464743 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:45.464711    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7efed674-a0ee-438e-aa0c-664a46b49c6b-kube-api-access-7w5lm" pod "7efed674-a0ee-438e-aa0c-664a46b49c6b" (UID: "7efed674-a0ee-438e-aa0c-664a46b49c6b"). InnerVolumeSpecName "kube-api-access-7w5lm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:22:45.564212 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:45.564185    4477 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7efed674-a0ee-438e-aa0c-664a46b49c6b-maas-api-openshift-ca\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:22:45.564212 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:45.564214    4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7w5lm\" (UniqueName: \"kubernetes.io/projected/7efed674-a0ee-438e-aa0c-664a46b49c6b-kube-api-access-7w5lm\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:22:45.564346 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:45.564229    4477 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7efed674-a0ee-438e-aa0c-664a46b49c6b-maas-api-xks-ca\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:22:46.198206 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:46.198163    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cad8417b-6lnzr"
Oct 30 22:22:46.201805 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:46.201767    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cad8417b-6lnzr" event={"ID":"7efed674-a0ee-438e-aa0c-664a46b49c6b","Type":"ContainerDied","Data":"0c740d27f2e9e858ee689ff884ec380ce52a8b4b6e59c7771d33d9b674fd85f0"}
Oct 30 22:22:46.201998 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:46.201982    4477 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c740d27f2e9e858ee689ff884ec380ce52a8b4b6e59c7771d33d9b674fd85f0"
Oct 30 22:22:46.263876 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:46.263851    4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-p4vsn"
Oct 30 22:22:46.264254 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:46.264189    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-p4vsn"
Oct 30 22:22:46.264891 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:46.264871    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-p4vsn"
Oct 30 22:22:46.440711 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:46.440685    4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-rpk4x"
Oct 30 22:22:46.441159 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:46.441135    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-rpk4x"
Oct 30 22:22:46.441962 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:46.441939    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-rpk4x"
Oct 30 22:22:54.004258 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:54.004224    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f9e4062-hww42"]
Oct 30 22:22:54.005105 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:54.005083    4477 cpu_manager.go:414] "containerMap: removing container" podUID="7efed674-a0ee-438e-aa0c-664a46b49c6b" containerName="revoke-keys"
Oct 30 22:22:54.005105 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:54.005104    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7efed674-a0ee-438e-aa0c-664a46b49c6b" containerName="revoke-keys"
Oct 30 22:22:54.005303 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:54.005283    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="7efed674-a0ee-438e-aa0c-664a46b49c6b" containerName="revoke-keys"
Oct 30 22:22:54.009085 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:54.009066    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f9e4062-hww42"
Oct 30 22:22:54.035566 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:54.035547    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:22:54.035688 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:54.035671    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 22:22:54.041488 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:54.041461    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f9e4062-hww42"]
Oct 30 22:22:54.191882 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:54.191841    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3f10f78-1693-47a0-9f53-eadb243ca6d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f9e4062-hww42\" (UID: \"f3f10f78-1693-47a0-9f53-eadb243ca6d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f9e4062-hww42"
Oct 30 22:22:54.192074 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:54.191927    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3f10f78-1693-47a0-9f53-eadb243ca6d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f9e4062-hww42\" (UID: \"f3f10f78-1693-47a0-9f53-eadb243ca6d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f9e4062-hww42"
Oct 30 22:22:54.192194 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:54.192176    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bv6kz\" (UniqueName: \"kubernetes.io/projected/f3f10f78-1693-47a0-9f53-eadb243ca6d8-kube-api-access-bv6kz\") pod \"maas-api-revoke-sub-9f9e4062-hww42\" (UID: \"f3f10f78-1693-47a0-9f53-eadb243ca6d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f9e4062-hww42"
Oct 30 22:22:54.293541 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:54.293452    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bv6kz\" (UniqueName: \"kubernetes.io/projected/f3f10f78-1693-47a0-9f53-eadb243ca6d8-kube-api-access-bv6kz\") pod \"maas-api-revoke-sub-9f9e4062-hww42\" (UID: \"f3f10f78-1693-47a0-9f53-eadb243ca6d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f9e4062-hww42"
Oct 30 22:22:54.293732 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:54.293582    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3f10f78-1693-47a0-9f53-eadb243ca6d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f9e4062-hww42\" (UID: \"f3f10f78-1693-47a0-9f53-eadb243ca6d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f9e4062-hww42"
Oct 30 22:22:54.293732 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:54.293631    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3f10f78-1693-47a0-9f53-eadb243ca6d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f9e4062-hww42\" (UID: \"f3f10f78-1693-47a0-9f53-eadb243ca6d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f9e4062-hww42"
Oct 30 22:22:54.294653 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:54.294623    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3f10f78-1693-47a0-9f53-eadb243ca6d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f9e4062-hww42\" (UID: \"f3f10f78-1693-47a0-9f53-eadb243ca6d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f9e4062-hww42"
Oct 30 22:22:54.296455 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:54.296427    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3f10f78-1693-47a0-9f53-eadb243ca6d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f9e4062-hww42\" (UID: \"f3f10f78-1693-47a0-9f53-eadb243ca6d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f9e4062-hww42"
Oct 30 22:22:54.312111 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:54.312085    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bv6kz\" (UniqueName: \"kubernetes.io/projected/f3f10f78-1693-47a0-9f53-eadb243ca6d8-kube-api-access-bv6kz\") pod \"maas-api-revoke-sub-9f9e4062-hww42\" (UID: \"f3f10f78-1693-47a0-9f53-eadb243ca6d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f9e4062-hww42"
Oct 30 22:22:54.322930 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:54.322906    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f9e4062-hww42"
Oct 30 22:22:54.636070 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:54.636020    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f9e4062-hww42"]
Oct 30 22:22:54.639578 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:22:54.639550    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3f10f78_1693_47a0_9f53_eadb243ca6d8.slice/crio-b02e26fee5fffeba3f93a88c7577e81d5275fc10e0853ce7c648d5f95540c5a8 WatchSource:0}: Error finding container b02e26fee5fffeba3f93a88c7577e81d5275fc10e0853ce7c648d5f95540c5a8: Status 404 returned error can't find the container with id b02e26fee5fffeba3f93a88c7577e81d5275fc10e0853ce7c648d5f95540c5a8
Oct 30 22:22:55.254985 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:55.254906    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f9e4062-hww42" event={"ID":"f3f10f78-1693-47a0-9f53-eadb243ca6d8","Type":"ContainerStarted","Data":"235ceba0e4b32f16deec3aa16da4fc194e289058686bee89f6a99ba7340e56b5"}
Oct 30 22:22:55.254985 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:55.254958    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f9e4062-hww42" event={"ID":"f3f10f78-1693-47a0-9f53-eadb243ca6d8","Type":"ContainerStarted","Data":"b02e26fee5fffeba3f93a88c7577e81d5275fc10e0853ce7c648d5f95540c5a8"}
Oct 30 22:22:55.293833 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:55.293783    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f9e4062-hww42" podStartSLOduration=2.293769745 podStartE2EDuration="2.293769745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:22:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:22:55.290198994 +0000 UTC m=+959.216125202" watchObservedRunningTime="2026-10-30 22:22:55.293769745 +0000 UTC m=+959.219695950"
Oct 30 22:22:56.264627 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:56.264587    4477 generic.go:356] "Generic (PLEG): container finished" podID="f3f10f78-1693-47a0-9f53-eadb243ca6d8" containerID="235ceba0e4b32f16deec3aa16da4fc194e289058686bee89f6a99ba7340e56b5" exitCode=0
Oct 30 22:22:56.265015 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:56.264680    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f9e4062-hww42" event={"ID":"f3f10f78-1693-47a0-9f53-eadb243ca6d8","Type":"ContainerDied","Data":"235ceba0e4b32f16deec3aa16da4fc194e289058686bee89f6a99ba7340e56b5"}
Oct 30 22:22:56.275241 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:22:56.275208    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 22:22:57.445015 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:57.444993    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f9e4062-hww42"
Oct 30 22:22:57.543199 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:57.543122    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3f10f78-1693-47a0-9f53-eadb243ca6d8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3f10f78-1693-47a0-9f53-eadb243ca6d8-maas-api-openshift-ca\") pod \"f3f10f78-1693-47a0-9f53-eadb243ca6d8\" (UID: \"f3f10f78-1693-47a0-9f53-eadb243ca6d8\") "
Oct 30 22:22:57.543199 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:57.543181    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3f10f78-1693-47a0-9f53-eadb243ca6d8-kube-api-access-bv6kz\" (UniqueName: \"kubernetes.io/projected/f3f10f78-1693-47a0-9f53-eadb243ca6d8-kube-api-access-bv6kz\") pod \"f3f10f78-1693-47a0-9f53-eadb243ca6d8\" (UID: \"f3f10f78-1693-47a0-9f53-eadb243ca6d8\") "
Oct 30 22:22:57.543446 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:57.543419    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3f10f78-1693-47a0-9f53-eadb243ca6d8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3f10f78-1693-47a0-9f53-eadb243ca6d8-maas-api-xks-ca\") pod \"f3f10f78-1693-47a0-9f53-eadb243ca6d8\" (UID: \"f3f10f78-1693-47a0-9f53-eadb243ca6d8\") "
Oct 30 22:22:57.543575 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:57.543542    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3f10f78-1693-47a0-9f53-eadb243ca6d8-maas-api-openshift-ca" pod "f3f10f78-1693-47a0-9f53-eadb243ca6d8" (UID: "f3f10f78-1693-47a0-9f53-eadb243ca6d8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:22:57.544113 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:57.544086    4477 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3f10f78-1693-47a0-9f53-eadb243ca6d8-maas-api-openshift-ca\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:22:57.545748 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:57.545694    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3f10f78-1693-47a0-9f53-eadb243ca6d8-maas-api-xks-ca" pod "f3f10f78-1693-47a0-9f53-eadb243ca6d8" (UID: "f3f10f78-1693-47a0-9f53-eadb243ca6d8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:22:57.545950 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:57.545923    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3f10f78-1693-47a0-9f53-eadb243ca6d8-kube-api-access-bv6kz" pod "f3f10f78-1693-47a0-9f53-eadb243ca6d8" (UID: "f3f10f78-1693-47a0-9f53-eadb243ca6d8"). InnerVolumeSpecName "kube-api-access-bv6kz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:22:57.577628 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:57.577599    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6bz92"]
Oct 30 22:22:57.645254 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:57.645219    4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bv6kz\" (UniqueName: \"kubernetes.io/projected/f3f10f78-1693-47a0-9f53-eadb243ca6d8-kube-api-access-bv6kz\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:22:57.645254 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:57.645247    4477 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3f10f78-1693-47a0-9f53-eadb243ca6d8-maas-api-xks-ca\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:22:58.279746 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:58.279717    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f9e4062-hww42"
Oct 30 22:22:58.279746 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:58.279725    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f9e4062-hww42" event={"ID":"f3f10f78-1693-47a0-9f53-eadb243ca6d8","Type":"ContainerDied","Data":"b02e26fee5fffeba3f93a88c7577e81d5275fc10e0853ce7c648d5f95540c5a8"}
Oct 30 22:22:58.279907 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:58.279765    4477 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b02e26fee5fffeba3f93a88c7577e81d5275fc10e0853ce7c648d5f95540c5a8"
Oct 30 22:22:59.807067 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:59.796129    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/simulator-w1-2b1e88-kserve-6c579b99f8-q5wpq" podUID="722f10d8-b7a2-4bf7-a274-9669529ef427" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:59.939065 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:59.938159    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0ad7c5/premium-w4-0ad7c5-kserve-7bbf78cd4-pm6wc" podUID="5c10371e-56e7-42ec-9ef8-0802e15cb2ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded"
Oct 30 22:22:59.961097 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:22:59.949106    4477 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-2-w4-0ad7c5-kserve-77c5467fcb-skspk" podUID="430bd879-2a89-4070-bb1b-9b9958a261b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": context deadline exceeded"
Oct 30 22:23:00.001079 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:00.000123    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/distinct-w2-421583-kserve-69cf5f4d6b-qbgnt" podUID="510ad10d-0554-47c8-9b55-deb359795071" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:00.026065 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:00.022171    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-2-w4-0ad7c5-kserve-77c5467fcb-skspk" podUID="430bd879-2a89-4070-bb1b-9b9958a261b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:00.049065 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:00.044179    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/distinct-2-w6-012ba0-kserve-78dbcd8c46-hqdfm" podUID="8326b014-989b-4722-befb-6fa8ea60688b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:00.135067 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:00.134128    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/distinct-w3-d52c59-kserve-ffb5f5b96-l58xl" podUID="c7b1483b-2631-4aa2-a888-3096761d5494" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:00.177067 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:00.176724    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/premium-w5-54114e-kserve-57f6747bcc-7wjrw" podUID="61990d22-ab8f-45bc-a4d6-4c8d528742f1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:00.255067 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:00.254167    4477 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-d52c59/distinct-w3-d52c59-kserve-ffb5f5b96-l58xl" podUID="c7b1483b-2631-4aa2-a888-3096761d5494" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:00.255067 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:00.254498    4477 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-421583/distinct-w2-421583-kserve-69cf5f4d6b-qbgnt" podUID="510ad10d-0554-47c8-9b55-deb359795071" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": context deadline exceeded"
Oct 30 22:23:00.287066 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:00.286581    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/premium-w6-012ba0-kserve-658654dbdb-d2k27" podUID="085847e5-4f50-419e-984f-b612b104ed90" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:00.475111 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:00.474900    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/distinct-2-w0-09d0d5-kserve-7dcf89b98b-7cjph" podUID="8cbd04b4-bb3e-4bbb-a41a-46131c9de051" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:00.803117 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:00.802200    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/simulator-w1-2b1e88-kserve-6c579b99f8-q5wpq" podUID="722f10d8-b7a2-4bf7-a274-9669529ef427" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded"
Oct 30 22:23:00.803299 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:00.803173    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-2b1e88/simulator-w1-2b1e88-kserve-6c579b99f8-q5wpq"
Oct 30 22:23:00.957062 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:00.954161    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0ad7c5/premium-w4-0ad7c5-kserve-7bbf78cd4-pm6wc" podUID="5c10371e-56e7-42ec-9ef8-0802e15cb2ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:00.957062 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:00.954964    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0ad7c5/premium-w4-0ad7c5-kserve-7bbf78cd4-pm6wc"
Oct 30 22:23:01.030068 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:01.025442    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-2-w4-0ad7c5-kserve-77c5467fcb-skspk" podUID="430bd879-2a89-4070-bb1b-9b9958a261b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:01.030068 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:01.027544    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-2-w4-0ad7c5-kserve-77c5467fcb-skspk"
Oct 30 22:23:01.030068 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:01.029611    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/distinct-w2-421583-kserve-69cf5f4d6b-qbgnt" podUID="510ad10d-0554-47c8-9b55-deb359795071" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:01.030068 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:01.029912    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-421583/distinct-w2-421583-kserve-69cf5f4d6b-qbgnt"
Oct 30 22:23:01.130602 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:01.060571    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/distinct-2-w6-012ba0-kserve-78dbcd8c46-hqdfm" podUID="8326b014-989b-4722-befb-6fa8ea60688b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:01.130602 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:01.069428    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-012ba0/distinct-2-w6-012ba0-kserve-78dbcd8c46-hqdfm"
Oct 30 22:23:01.223068 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:01.154127    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/distinct-w3-d52c59-kserve-ffb5f5b96-l58xl" podUID="c7b1483b-2631-4aa2-a888-3096761d5494" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:01.223068 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:01.154571    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d52c59/distinct-w3-d52c59-kserve-ffb5f5b96-l58xl"
Oct 30 22:23:01.223068 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:01.200230    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/premium-w5-54114e-kserve-57f6747bcc-7wjrw" podUID="61990d22-ab8f-45bc-a4d6-4c8d528742f1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:01.223068 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:01.200683    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-54114e/premium-w5-54114e-kserve-57f6747bcc-7wjrw"
Oct 30 22:23:01.321029 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:01.320328    4477 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0ad7c5/premium-w4-0ad7c5-kserve-7bbf78cd4-pm6wc" podUID="5c10371e-56e7-42ec-9ef8-0802e15cb2ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:01.444156 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:01.438823    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/premium-w6-012ba0-kserve-658654dbdb-d2k27" podUID="085847e5-4f50-419e-984f-b612b104ed90" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:01.444156 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:01.439331    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-012ba0/premium-w6-012ba0-kserve-658654dbdb-d2k27"
Oct 30 22:23:01.496062 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:01.492157    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/distinct-2-w0-09d0d5-kserve-7dcf89b98b-7cjph" podUID="8cbd04b4-bb3e-4bbb-a41a-46131c9de051" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:01.496062 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:01.492771    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-09d0d5/distinct-2-w0-09d0d5-kserve-7dcf89b98b-7cjph"
Oct 30 22:23:01.564066 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:01.546134    4477 patch_prober.go:29] interesting pod/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-cjkc9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:23:01.564066 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:01.546193    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-cjkc9" podUID="e8a3aba3-7d06-4fd5-8106-2d02250f1da2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:01.821133 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:01.819967    4477 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-54114e/premium-w5-54114e-kserve-57f6747bcc-7wjrw" podUID="61990d22-ab8f-45bc-a4d6-4c8d528742f1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/health\": context deadline exceeded"
Oct 30 22:23:01.879247 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:01.827370    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/simulator-w1-2b1e88-kserve-6c579b99f8-q5wpq" podUID="722f10d8-b7a2-4bf7-a274-9669529ef427" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:02.000631 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:01.966133    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0ad7c5/premium-w4-0ad7c5-kserve-7bbf78cd4-pm6wc" podUID="5c10371e-56e7-42ec-9ef8-0802e15cb2ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded"
Oct 30 22:23:02.035020 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:02.034972    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/distinct-w2-421583-kserve-69cf5f4d6b-qbgnt" podUID="510ad10d-0554-47c8-9b55-deb359795071" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Oct 30 22:23:02.096161 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:02.077135    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-2-w4-0ad7c5-kserve-77c5467fcb-skspk" podUID="430bd879-2a89-4070-bb1b-9b9958a261b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:02.163082 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:02.159183    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/distinct-w3-d52c59-kserve-ffb5f5b96-l58xl" podUID="c7b1483b-2631-4aa2-a888-3096761d5494" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:02.182372 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:02.182169    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/distinct-2-w6-012ba0-kserve-78dbcd8c46-hqdfm" podUID="8326b014-989b-4722-befb-6fa8ea60688b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:02.215511 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:02.207427    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/premium-w5-54114e-kserve-57f6747bcc-7wjrw" podUID="61990d22-ab8f-45bc-a4d6-4c8d528742f1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded"
Oct 30 22:23:02.273131 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:02.265483    4477 patch_prober.go:29] interesting pod/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hq9vz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:23:02.273131 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:02.265545    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hq9vz" podUID="76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:02.464063 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:02.439120    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/premium-w6-012ba0-kserve-658654dbdb-d2k27" podUID="085847e5-4f50-419e-984f-b612b104ed90" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:02.517545 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:02.517489    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/distinct-2-w0-09d0d5-kserve-7dcf89b98b-7cjph" podUID="8cbd04b4-bb3e-4bbb-a41a-46131c9de051" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:02.561216 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:02.561180    4477 patch_prober.go:29] interesting pod/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-rpk4x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:23:02.561488 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:02.561445    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-rpk4x" podUID="69b42852-cbc4-46b7-be1b-98ce37efa0d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:02.708494 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:02.708432    4477 patch_prober.go:29] interesting pod/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-9j5tr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:23:02.708792 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:02.708743    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-9j5tr" podUID="1fb811da-4dc2-48bb-bec6-afb7a1a80ed2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:02.829389 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:02.829243    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/simulator-w1-2b1e88-kserve-6c579b99f8-q5wpq" podUID="722f10d8-b7a2-4bf7-a274-9669529ef427" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded"
Oct 30 22:23:02.860098 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:02.842174    4477 patch_prober.go:29] interesting pod/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-z7ggc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:23:02.860098 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:02.842241    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-z7ggc" podUID="5779d15a-20b4-4ec0-ac5e-b8bdfa89096d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:02.895219 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:02.884287    4477 patch_prober.go:29] interesting pod/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-m74tl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:23:02.895219 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:02.884355    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-m74tl" podUID="3e6f6850-1ca2-4f4c-a047-23e35ce2b697" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:02.972065 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:02.969132    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0ad7c5/premium-w4-0ad7c5-kserve-7bbf78cd4-pm6wc" podUID="5c10371e-56e7-42ec-9ef8-0802e15cb2ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:02.983683 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:02.983552    4477 patch_prober.go:29] interesting pod/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-sqxvr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:23:02.983683 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:02.983611    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-sqxvr" podUID="909907e9-319a-48dc-90d6-735cee614217" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:03.046564 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:03.046526    4477 patch_prober.go:29] interesting pod/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-vtwp6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:23:03.047182 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:03.047147    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-vtwp6" podUID="623d861e-ea33-4cd5-af9e-3b29b937e335" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:03.047598 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:03.047569    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/distinct-w2-421583-kserve-69cf5f4d6b-qbgnt" podUID="510ad10d-0554-47c8-9b55-deb359795071" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:03.098422 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:03.088055    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-2-w4-0ad7c5-kserve-77c5467fcb-skspk" podUID="430bd879-2a89-4070-bb1b-9b9958a261b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Oct 30 22:23:03.181509 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:03.181458    4477 patch_prober.go:29] interesting pod/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-lg99w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:23:03.181787 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:03.181739    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-lg99w" podUID="8e7915fd-e76a-407d-825e-ae095f5d3c56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:03.222529 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:03.222485    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/distinct-2-w6-012ba0-kserve-78dbcd8c46-hqdfm" podUID="8326b014-989b-4722-befb-6fa8ea60688b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:03.234829 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:03.233356    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-2b1e88/simulator-w1-2b1e88-kserve-6c579b99f8-q5wpq"
Oct 30 22:23:03.265595 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:03.263734    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/distinct-w3-d52c59-kserve-ffb5f5b96-l58xl" podUID="c7b1483b-2631-4aa2-a888-3096761d5494" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:03.266675 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:03.222202    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/premium-w5-54114e-kserve-57f6747bcc-7wjrw" podUID="61990d22-ab8f-45bc-a4d6-4c8d528742f1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:03.442170 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:03.442121    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/premium-w6-012ba0-kserve-658654dbdb-d2k27" podUID="085847e5-4f50-419e-984f-b612b104ed90" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:03.572068 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:03.566167    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/distinct-2-w0-09d0d5-kserve-7dcf89b98b-7cjph" podUID="8cbd04b4-bb3e-4bbb-a41a-46131c9de051" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Oct 30 22:23:03.980650 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:03.979880    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0ad7c5/premium-w4-0ad7c5-kserve-7bbf78cd4-pm6wc" podUID="5c10371e-56e7-42ec-9ef8-0802e15cb2ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded"
Oct 30 22:23:04.054658 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:04.048979    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/distinct-w2-421583-kserve-69cf5f4d6b-qbgnt" podUID="510ad10d-0554-47c8-9b55-deb359795071" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:04.105065 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:04.101136    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-2-w4-0ad7c5-kserve-77c5467fcb-skspk" podUID="430bd879-2a89-4070-bb1b-9b9958a261b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:04.225066 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:04.224694    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/distinct-2-w6-012ba0-kserve-78dbcd8c46-hqdfm" podUID="8326b014-989b-4722-befb-6fa8ea60688b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:04.273288 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:04.273145    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/distinct-w3-d52c59-kserve-ffb5f5b96-l58xl" podUID="c7b1483b-2631-4aa2-a888-3096761d5494" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:04.276510 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:04.276477    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/premium-w5-54114e-kserve-57f6747bcc-7wjrw" podUID="61990d22-ab8f-45bc-a4d6-4c8d528742f1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded"
Oct 30 22:23:04.513066 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:04.467451    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/premium-w6-012ba0-kserve-658654dbdb-d2k27" podUID="085847e5-4f50-419e-984f-b612b104ed90" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:04.577984 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:04.577827    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/distinct-2-w0-09d0d5-kserve-7dcf89b98b-7cjph" podUID="8cbd04b4-bb3e-4bbb-a41a-46131c9de051" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:04.869237 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:04.869126    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/simulator-w1-2b1e88-kserve-6c579b99f8-q5wpq" podUID="722f10d8-b7a2-4bf7-a274-9669529ef427" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded"
Oct 30 22:23:04.990962 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:04.990922    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0ad7c5/premium-w4-0ad7c5-kserve-7bbf78cd4-pm6wc" podUID="5c10371e-56e7-42ec-9ef8-0802e15cb2ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:05.020150 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:05.020109    4477 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-2b1e88/simulator-w1-2b1e88-kserve-6c579b99f8-q5wpq" podUID="722f10d8-b7a2-4bf7-a274-9669529ef427" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": context deadline exceeded"
Oct 30 22:23:05.050679 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:05.050620    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/distinct-w2-421583-kserve-69cf5f4d6b-qbgnt" podUID="510ad10d-0554-47c8-9b55-deb359795071" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:05.128231 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:05.110118    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-2-w4-0ad7c5-kserve-77c5467fcb-skspk" podUID="430bd879-2a89-4070-bb1b-9b9958a261b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:05.162380 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:05.161354    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d52c59/distinct-w3-d52c59-kserve-ffb5f5b96-l58xl"
Oct 30 22:23:05.170020 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:05.169999    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-421583/distinct-w2-421583-kserve-69cf5f4d6b-qbgnt"
Oct 30 22:23:05.300263 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:05.300225    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/premium-w5-54114e-kserve-57f6747bcc-7wjrw" podUID="61990d22-ab8f-45bc-a4d6-4c8d528742f1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded"
Oct 30 22:23:05.300872 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:05.300841    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/distinct-2-w6-012ba0-kserve-78dbcd8c46-hqdfm" podUID="8326b014-989b-4722-befb-6fa8ea60688b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Oct 30 22:23:05.401543 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:05.401467    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-54114e/premium-w5-54114e-kserve-57f6747bcc-7wjrw"
Oct 30 22:23:05.418694 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:05.409635    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-2-w4-0ad7c5-kserve-77c5467fcb-skspk"
Oct 30 22:23:05.485220 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:05.454255    4477 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-09d0d5/distinct-2-w0-09d0d5-kserve-7dcf89b98b-7cjph" podUID="8cbd04b4-bb3e-4bbb-a41a-46131c9de051" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:05.528543 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:05.528507    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-012ba0/premium-w6-012ba0-kserve-658654dbdb-d2k27"
Oct 30 22:23:05.640261 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:05.599635    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/distinct-2-w0-09d0d5-kserve-7dcf89b98b-7cjph" podUID="8cbd04b4-bb3e-4bbb-a41a-46131c9de051" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Oct 30 22:23:05.640745 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:05.616442    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0ad7c5/premium-w4-0ad7c5-kserve-7bbf78cd4-pm6wc"
Oct 30 22:23:05.679793 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:05.679722    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-012ba0/distinct-2-w6-012ba0-kserve-78dbcd8c46-hqdfm"
Oct 30 22:23:05.680017 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:05.680002    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-09d0d5/distinct-2-w0-09d0d5-kserve-7dcf89b98b-7cjph"
Oct 30 22:23:06.267328 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:06.267304    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-220e38c3-7cx7g"]
Oct 30 22:23:06.268025 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:06.268008    4477 cpu_manager.go:414] "containerMap: removing container" podUID="f3f10f78-1693-47a0-9f53-eadb243ca6d8" containerName="revoke-keys"
Oct 30 22:23:06.268025 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:06.268025    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3f10f78-1693-47a0-9f53-eadb243ca6d8" containerName="revoke-keys"
Oct 30 22:23:06.268191 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:06.268180    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3f10f78-1693-47a0-9f53-eadb243ca6d8" containerName="revoke-keys"
Oct 30 22:23:06.271695 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:06.271677    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-220e38c3-7cx7g"
Oct 30 22:23:06.290997 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:06.290975    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-220e38c3-7cx7g"]
Oct 30 22:23:06.297311 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:06.296965    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:23:06.297311 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:06.297129    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 22:23:06.320275 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:06.320247    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7f67f\" (UniqueName: \"kubernetes.io/projected/ae8f9b4b-3744-40f2-a157-6ccb78a0151f-kube-api-access-7f67f\") pod \"maas-api-revoke-sub-220e38c3-7cx7g\" (UID: \"ae8f9b4b-3744-40f2-a157-6ccb78a0151f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-220e38c3-7cx7g"
Oct 30 22:23:06.320401 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:06.320355    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae8f9b4b-3744-40f2-a157-6ccb78a0151f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-220e38c3-7cx7g\" (UID: \"ae8f9b4b-3744-40f2-a157-6ccb78a0151f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-220e38c3-7cx7g"
Oct 30 22:23:06.320996 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:06.320471    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae8f9b4b-3744-40f2-a157-6ccb78a0151f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-220e38c3-7cx7g\" (UID: \"ae8f9b4b-3744-40f2-a157-6ccb78a0151f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-220e38c3-7cx7g"
Oct 30 22:23:06.421423 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:06.421394    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae8f9b4b-3744-40f2-a157-6ccb78a0151f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-220e38c3-7cx7g\" (UID: \"ae8f9b4b-3744-40f2-a157-6ccb78a0151f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-220e38c3-7cx7g"
Oct 30 22:23:06.421572 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:06.421483    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae8f9b4b-3744-40f2-a157-6ccb78a0151f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-220e38c3-7cx7g\" (UID: \"ae8f9b4b-3744-40f2-a157-6ccb78a0151f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-220e38c3-7cx7g"
Oct 30 22:23:06.421700 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:06.421682    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7f67f\" (UniqueName: \"kubernetes.io/projected/ae8f9b4b-3744-40f2-a157-6ccb78a0151f-kube-api-access-7f67f\") pod \"maas-api-revoke-sub-220e38c3-7cx7g\" (UID: \"ae8f9b4b-3744-40f2-a157-6ccb78a0151f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-220e38c3-7cx7g"
Oct 30 22:23:06.422123 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:06.422095    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae8f9b4b-3744-40f2-a157-6ccb78a0151f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-220e38c3-7cx7g\" (UID: \"ae8f9b4b-3744-40f2-a157-6ccb78a0151f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-220e38c3-7cx7g"
Oct 30 22:23:06.423747 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:06.423726    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae8f9b4b-3744-40f2-a157-6ccb78a0151f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-220e38c3-7cx7g\" (UID: \"ae8f9b4b-3744-40f2-a157-6ccb78a0151f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-220e38c3-7cx7g"
Oct 30 22:23:06.439562 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:06.439533    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7f67f\" (UniqueName: \"kubernetes.io/projected/ae8f9b4b-3744-40f2-a157-6ccb78a0151f-kube-api-access-7f67f\") pod \"maas-api-revoke-sub-220e38c3-7cx7g\" (UID: \"ae8f9b4b-3744-40f2-a157-6ccb78a0151f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-220e38c3-7cx7g"
Oct 30 22:23:06.586870 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:06.586811    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-220e38c3-7cx7g"
Oct 30 22:23:06.888220 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:06.888180    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-220e38c3-7cx7g"]
Oct 30 22:23:06.890911 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:23:06.890881    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae8f9b4b_3744_40f2_a157_6ccb78a0151f.slice/crio-888417ebdf7bc2f9185feacdf18e9326a97aeb383cc971483a09ef054a2c2957 WatchSource:0}: Error finding container 888417ebdf7bc2f9185feacdf18e9326a97aeb383cc971483a09ef054a2c2957: Status 404 returned error can't find the container with id 888417ebdf7bc2f9185feacdf18e9326a97aeb383cc971483a09ef054a2c2957
Oct 30 22:23:07.418600 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:07.418562    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-220e38c3-7cx7g" event={"ID":"ae8f9b4b-3744-40f2-a157-6ccb78a0151f","Type":"ContainerStarted","Data":"00f829ab61cf84bbb84b023f08995516af72f3787df1d7f4370ea62282ac3ec8"}
Oct 30 22:23:07.419109 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:07.418626    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-220e38c3-7cx7g" event={"ID":"ae8f9b4b-3744-40f2-a157-6ccb78a0151f","Type":"ContainerStarted","Data":"888417ebdf7bc2f9185feacdf18e9326a97aeb383cc971483a09ef054a2c2957"}
Oct 30 22:23:07.456828 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:07.456749    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-220e38c3-7cx7g" podStartSLOduration=1.456736251 podStartE2EDuration="1.456736251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:23:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:23:07.453461176 +0000 UTC m=+971.379387401" watchObservedRunningTime="2026-10-30 22:23:07.456736251 +0000 UTC m=+971.382662456"
Oct 30 22:23:08.429567 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:08.429531    4477 generic.go:356] "Generic (PLEG): container finished" podID="ae8f9b4b-3744-40f2-a157-6ccb78a0151f" containerID="00f829ab61cf84bbb84b023f08995516af72f3787df1d7f4370ea62282ac3ec8" exitCode=0
Oct 30 22:23:08.430075 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:08.429701    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-220e38c3-7cx7g" event={"ID":"ae8f9b4b-3744-40f2-a157-6ccb78a0151f","Type":"ContainerDied","Data":"00f829ab61cf84bbb84b023f08995516af72f3787df1d7f4370ea62282ac3ec8"}
Oct 30 22:23:09.608870 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:09.608843    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-220e38c3-7cx7g"
Oct 30 22:23:09.668576 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:09.668547    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae8f9b4b-3744-40f2-a157-6ccb78a0151f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae8f9b4b-3744-40f2-a157-6ccb78a0151f-maas-api-openshift-ca\") pod \"ae8f9b4b-3744-40f2-a157-6ccb78a0151f\" (UID: \"ae8f9b4b-3744-40f2-a157-6ccb78a0151f\") "
Oct 30 22:23:09.668717 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:09.668599    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae8f9b4b-3744-40f2-a157-6ccb78a0151f-kube-api-access-7f67f\" (UniqueName: \"kubernetes.io/projected/ae8f9b4b-3744-40f2-a157-6ccb78a0151f-kube-api-access-7f67f\") pod \"ae8f9b4b-3744-40f2-a157-6ccb78a0151f\" (UID: \"ae8f9b4b-3744-40f2-a157-6ccb78a0151f\") "
Oct 30 22:23:09.668765 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:09.668711    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae8f9b4b-3744-40f2-a157-6ccb78a0151f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae8f9b4b-3744-40f2-a157-6ccb78a0151f-maas-api-xks-ca\") pod \"ae8f9b4b-3744-40f2-a157-6ccb78a0151f\" (UID: \"ae8f9b4b-3744-40f2-a157-6ccb78a0151f\") "
Oct 30 22:23:09.669028 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:09.668998    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae8f9b4b-3744-40f2-a157-6ccb78a0151f-maas-api-openshift-ca" pod "ae8f9b4b-3744-40f2-a157-6ccb78a0151f" (UID: "ae8f9b4b-3744-40f2-a157-6ccb78a0151f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:23:09.669382 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:09.669355    4477 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae8f9b4b-3744-40f2-a157-6ccb78a0151f-maas-api-openshift-ca\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:23:09.670836 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:09.670808    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae8f9b4b-3744-40f2-a157-6ccb78a0151f-maas-api-xks-ca" pod "ae8f9b4b-3744-40f2-a157-6ccb78a0151f" (UID: "ae8f9b4b-3744-40f2-a157-6ccb78a0151f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:23:09.671377 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:09.671350    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae8f9b4b-3744-40f2-a157-6ccb78a0151f-kube-api-access-7f67f" pod "ae8f9b4b-3744-40f2-a157-6ccb78a0151f" (UID: "ae8f9b4b-3744-40f2-a157-6ccb78a0151f"). InnerVolumeSpecName "kube-api-access-7f67f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:23:09.770263 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:09.770200    4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7f67f\" (UniqueName: \"kubernetes.io/projected/ae8f9b4b-3744-40f2-a157-6ccb78a0151f-kube-api-access-7f67f\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:23:09.770263 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:09.770228    4477 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae8f9b4b-3744-40f2-a157-6ccb78a0151f-maas-api-xks-ca\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:23:10.447480 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:10.447430    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-220e38c3-7cx7g"
Oct 30 22:23:10.447480 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:10.447460    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-220e38c3-7cx7g" event={"ID":"ae8f9b4b-3744-40f2-a157-6ccb78a0151f","Type":"ContainerDied","Data":"888417ebdf7bc2f9185feacdf18e9326a97aeb383cc971483a09ef054a2c2957"}
Oct 30 22:23:10.447684 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:10.447519    4477 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="888417ebdf7bc2f9185feacdf18e9326a97aeb383cc971483a09ef054a2c2957"
Oct 30 22:23:12.303826 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:12.303797    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7278-2b5f86"]
Oct 30 22:23:12.304419 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:12.304404    4477 cpu_manager.go:414] "containerMap: removing container" podUID="ae8f9b4b-3744-40f2-a157-6ccb78a0151f" containerName="revoke-keys"
Oct 30 22:23:12.304462 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:12.304421    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae8f9b4b-3744-40f2-a157-6ccb78a0151f" containerName="revoke-keys"
Oct 30 22:23:12.304544 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:12.304534    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae8f9b4b-3744-40f2-a157-6ccb78a0151f" containerName="revoke-keys"
Oct 30 22:23:12.308052 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:12.308024    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7278-2b5f86"
Oct 30 22:23:12.322151 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:12.322128    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-xbw52\""
Oct 30 22:23:12.323486 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:12.323466    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7278-2b5f86"]
Oct 30 22:23:12.401248 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:12.401216    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7ttc\" (UniqueName: \"kubernetes.io/projected/9ae92182-d83c-4955-88b8-0524f3c41b00-kube-api-access-s7ttc\") pod \"test-curl-7278-2b5f86\" (UID: \"9ae92182-d83c-4955-88b8-0524f3c41b00\") " pod="opendatahub/test-curl-7278-2b5f86"
Oct 30 22:23:12.502333 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:12.502303    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7ttc\" (UniqueName: \"kubernetes.io/projected/9ae92182-d83c-4955-88b8-0524f3c41b00-kube-api-access-s7ttc\") pod \"test-curl-7278-2b5f86\" (UID: \"9ae92182-d83c-4955-88b8-0524f3c41b00\") " pod="opendatahub/test-curl-7278-2b5f86"
Oct 30 22:23:12.520536 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:12.520500    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7ttc\" (UniqueName: \"kubernetes.io/projected/9ae92182-d83c-4955-88b8-0524f3c41b00-kube-api-access-s7ttc\") pod \"test-curl-7278-2b5f86\" (UID: \"9ae92182-d83c-4955-88b8-0524f3c41b00\") " pod="opendatahub/test-curl-7278-2b5f86"
Oct 30 22:23:12.620083 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:12.619996    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7278-2b5f86"
Oct 30 22:23:12.921612 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:23:12.921571    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9ae92182_d83c_4955_88b8_0524f3c41b00.slice/crio-290af3d98cb88aae0a745ec77d49c5098d778587fd90fe9fb176b01aa0ab6824 WatchSource:0}: Error finding container 290af3d98cb88aae0a745ec77d49c5098d778587fd90fe9fb176b01aa0ab6824: Status 404 returned error can't find the container with id 290af3d98cb88aae0a745ec77d49c5098d778587fd90fe9fb176b01aa0ab6824
Oct 30 22:23:12.922737 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:12.922712    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7278-2b5f86"]
Oct 30 22:23:12.923820 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:12.923802    4477 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 22:23:13.468368 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:13.468340    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7278-2b5f86" event={"ID":"9ae92182-d83c-4955-88b8-0524f3c41b00","Type":"ContainerStarted","Data":"290af3d98cb88aae0a745ec77d49c5098d778587fd90fe9fb176b01aa0ab6824"}
Oct 30 22:23:14.476884 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:14.476847    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7278-2b5f86" event={"ID":"9ae92182-d83c-4955-88b8-0524f3c41b00","Type":"ContainerStarted","Data":"23183ea6d7f427856e65b865517c2ea3c36de8dfcadf138ffc90ef10e30a8a9c"}
Oct 30 22:23:14.512021 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:14.511973    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7278-2b5f86" podStartSLOduration=1.990905455 podStartE2EDuration="2.511918935s" totalImagesPullingTime="521.01348ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:23:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:23:14.510818016 +0000 UTC m=+978.436744224" watchObservedRunningTime="2026-10-30 22:23:14.511918935 +0000 UTC m=+978.437845141"
Oct 30 22:23:15.381965 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:15.381897    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7278-2b5f86"]
Oct 30 22:23:15.385580 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:15.385555    4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7278-2b5f86"]
Oct 30 22:23:15.545377 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:15.545348    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7278-ea2049"]
Oct 30 22:23:15.545924 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:15.545911    4477 cpu_manager.go:414] "containerMap: removing container" podUID="9ae92182-d83c-4955-88b8-0524f3c41b00" containerName="test-curl-7278-2b5f86"
Oct 30 22:23:15.545924 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:15.545925    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ae92182-d83c-4955-88b8-0524f3c41b00" containerName="test-curl-7278-2b5f86"
Oct 30 22:23:15.546083 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:15.546073    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ae92182-d83c-4955-88b8-0524f3c41b00" containerName="test-curl-7278-2b5f86"
Oct 30 22:23:15.549538 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:15.549521    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7278-ea2049"
Oct 30 22:23:15.568932 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:15.568897    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7278-ea2049"]
Oct 30 22:23:15.646822 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:15.646756    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7xwn\" (UniqueName: \"kubernetes.io/projected/ed1fed68-255c-4925-9d03-8e1b638539e6-kube-api-access-q7xwn\") pod \"test-curl-7278-ea2049\" (UID: \"ed1fed68-255c-4925-9d03-8e1b638539e6\") " pod="opendatahub/test-curl-7278-ea2049"
Oct 30 22:23:15.748719 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:15.748684    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7xwn\" (UniqueName: \"kubernetes.io/projected/ed1fed68-255c-4925-9d03-8e1b638539e6-kube-api-access-q7xwn\") pod \"test-curl-7278-ea2049\" (UID: \"ed1fed68-255c-4925-9d03-8e1b638539e6\") " pod="opendatahub/test-curl-7278-ea2049"
Oct 30 22:23:15.766850 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:15.766827    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7xwn\" (UniqueName: \"kubernetes.io/projected/ed1fed68-255c-4925-9d03-8e1b638539e6-kube-api-access-q7xwn\") pod \"test-curl-7278-ea2049\" (UID: \"ed1fed68-255c-4925-9d03-8e1b638539e6\") " pod="opendatahub/test-curl-7278-ea2049"
Oct 30 22:23:15.862322 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:15.862301    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7278-ea2049"
Oct 30 22:23:16.157438 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:16.157411    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7278-ea2049"]
Oct 30 22:23:16.160745 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:23:16.160718    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poded1fed68_255c_4925_9d03_8e1b638539e6.slice/crio-8ac0ea50de21cc1cf19fb93fd53160dbbb67af8f12050672e2c760ff4d00f9cf WatchSource:0}: Error finding container 8ac0ea50de21cc1cf19fb93fd53160dbbb67af8f12050672e2c760ff4d00f9cf: Status 404 returned error can't find the container with id 8ac0ea50de21cc1cf19fb93fd53160dbbb67af8f12050672e2c760ff4d00f9cf
Oct 30 22:23:16.493528 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:16.493387    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7278-ea2049" event={"ID":"ed1fed68-255c-4925-9d03-8e1b638539e6","Type":"ContainerStarted","Data":"86b26ad149c6d9ceed05efeafb60f9c4bbc9a36a69db957965614d23a6c4fcbe"}
Oct 30 22:23:16.493528 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:16.493434    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7278-ea2049" event={"ID":"ed1fed68-255c-4925-9d03-8e1b638539e6","Type":"ContainerStarted","Data":"8ac0ea50de21cc1cf19fb93fd53160dbbb67af8f12050672e2c760ff4d00f9cf"}
Oct 30 22:23:16.493528 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:16.493471    4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7278-2b5f86" podUID="9ae92182-d83c-4955-88b8-0524f3c41b00" containerName="test-curl-7278-2b5f86" containerID="cri-o://23183ea6d7f427856e65b865517c2ea3c36de8dfcadf138ffc90ef10e30a8a9c" gracePeriod=2
Oct 30 22:23:16.527590 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:16.527542    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7278-ea2049" podStartSLOduration=1.527526846 podStartE2EDuration="1.527526846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:23:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:23:16.527338516 +0000 UTC m=+980.453264722" watchObservedRunningTime="2026-10-30 22:23:16.527526846 +0000 UTC m=+980.453453051"
Oct 30 22:23:17.394210 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:17.394174    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7278-ea2049"]
Oct 30 22:23:17.399690 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:17.399647    4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7278-ea2049"]
Oct 30 22:23:18.508001 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:18.507965    4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7278-ea2049" podUID="ed1fed68-255c-4925-9d03-8e1b638539e6" containerName="test-curl-7278-ea2049" containerID="cri-o://86b26ad149c6d9ceed05efeafb60f9c4bbc9a36a69db957965614d23a6c4fcbe" gracePeriod=2
Oct 30 22:23:18.722386 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:18.722357    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7278-2b5f86"
Oct 30 22:23:18.889029 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:18.888993    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ae92182-d83c-4955-88b8-0524f3c41b00-kube-api-access-s7ttc\" (UniqueName: \"kubernetes.io/projected/9ae92182-d83c-4955-88b8-0524f3c41b00-kube-api-access-s7ttc\") pod \"9ae92182-d83c-4955-88b8-0524f3c41b00\" (UID: \"9ae92182-d83c-4955-88b8-0524f3c41b00\") "
Oct 30 22:23:18.891213 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:18.891182    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ae92182-d83c-4955-88b8-0524f3c41b00-kube-api-access-s7ttc" pod "9ae92182-d83c-4955-88b8-0524f3c41b00" (UID: "9ae92182-d83c-4955-88b8-0524f3c41b00"). InnerVolumeSpecName "kube-api-access-s7ttc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:23:18.990404 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:18.990375    4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7ttc\" (UniqueName: \"kubernetes.io/projected/9ae92182-d83c-4955-88b8-0524f3c41b00-kube-api-access-s7ttc\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:23:19.524948 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:19.523541    4477 generic.go:356] "Generic (PLEG): container finished" podID="9ae92182-d83c-4955-88b8-0524f3c41b00" containerID="23183ea6d7f427856e65b865517c2ea3c36de8dfcadf138ffc90ef10e30a8a9c" exitCode=137
Oct 30 22:23:19.524948 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:19.523917    4477 scope.go:122] "RemoveContainer" containerID="23183ea6d7f427856e65b865517c2ea3c36de8dfcadf138ffc90ef10e30a8a9c"
Oct 30 22:23:19.524948 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:19.524101    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7278-2b5f86"
Oct 30 22:23:19.572273 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:19.572230    4477 scope.go:122] "RemoveContainer" containerID="23183ea6d7f427856e65b865517c2ea3c36de8dfcadf138ffc90ef10e30a8a9c"
Oct 30 22:23:19.572858 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:23:19.572830    4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23183ea6d7f427856e65b865517c2ea3c36de8dfcadf138ffc90ef10e30a8a9c\": container with ID starting with 23183ea6d7f427856e65b865517c2ea3c36de8dfcadf138ffc90ef10e30a8a9c not found: ID does not exist" containerID="23183ea6d7f427856e65b865517c2ea3c36de8dfcadf138ffc90ef10e30a8a9c"
Oct 30 22:23:19.573011 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:19.572988    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23183ea6d7f427856e65b865517c2ea3c36de8dfcadf138ffc90ef10e30a8a9c"} err="failed to get container status \"23183ea6d7f427856e65b865517c2ea3c36de8dfcadf138ffc90ef10e30a8a9c\": rpc error: code = NotFound desc = could not find container \"23183ea6d7f427856e65b865517c2ea3c36de8dfcadf138ffc90ef10e30a8a9c\": container with ID starting with 23183ea6d7f427856e65b865517c2ea3c36de8dfcadf138ffc90ef10e30a8a9c not found: ID does not exist"
Oct 30 22:23:19.577256 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:23:19.577225    4477 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7278-2b5f86\" is forbidden: User \"system:node:ip-10-0-130-77.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-77.ec2.internal' and this object" podUID="9ae92182-d83c-4955-88b8-0524f3c41b00" pod="opendatahub/test-curl-7278-2b5f86"
Oct 30 22:23:20.243063 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:20.242727    4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ae92182-d83c-4955-88b8-0524f3c41b00" path="/var/lib/kubelet/pods/9ae92182-d83c-4955-88b8-0524f3c41b00/volumes"
Oct 30 22:23:20.829160 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:20.829134    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7278-ea2049"
Oct 30 22:23:20.857445 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:20.857414    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed1fed68-255c-4925-9d03-8e1b638539e6-kube-api-access-q7xwn\" (UniqueName: \"kubernetes.io/projected/ed1fed68-255c-4925-9d03-8e1b638539e6-kube-api-access-q7xwn\") pod \"ed1fed68-255c-4925-9d03-8e1b638539e6\" (UID: \"ed1fed68-255c-4925-9d03-8e1b638539e6\") "
Oct 30 22:23:20.859547 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:20.859519    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed1fed68-255c-4925-9d03-8e1b638539e6-kube-api-access-q7xwn" pod "ed1fed68-255c-4925-9d03-8e1b638539e6" (UID: "ed1fed68-255c-4925-9d03-8e1b638539e6"). InnerVolumeSpecName "kube-api-access-q7xwn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:23:20.960227 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:20.960195    4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7xwn\" (UniqueName: \"kubernetes.io/projected/ed1fed68-255c-4925-9d03-8e1b638539e6-kube-api-access-q7xwn\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:23:21.546107 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:21.542188    4477 generic.go:356] "Generic (PLEG): container finished" podID="ed1fed68-255c-4925-9d03-8e1b638539e6" containerID="86b26ad149c6d9ceed05efeafb60f9c4bbc9a36a69db957965614d23a6c4fcbe" exitCode=137
Oct 30 22:23:21.546107 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:21.542416    4477 scope.go:122] "RemoveContainer" containerID="86b26ad149c6d9ceed05efeafb60f9c4bbc9a36a69db957965614d23a6c4fcbe"
Oct 30 22:23:21.546107 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:21.542538    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7278-ea2049"
Oct 30 22:23:21.557428 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:21.557410    4477 scope.go:122] "RemoveContainer" containerID="86b26ad149c6d9ceed05efeafb60f9c4bbc9a36a69db957965614d23a6c4fcbe"
Oct 30 22:23:21.557697 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:23:21.557681    4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86b26ad149c6d9ceed05efeafb60f9c4bbc9a36a69db957965614d23a6c4fcbe\": container with ID starting with 86b26ad149c6d9ceed05efeafb60f9c4bbc9a36a69db957965614d23a6c4fcbe not found: ID does not exist" containerID="86b26ad149c6d9ceed05efeafb60f9c4bbc9a36a69db957965614d23a6c4fcbe"
Oct 30 22:23:21.557740 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:21.557703    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86b26ad149c6d9ceed05efeafb60f9c4bbc9a36a69db957965614d23a6c4fcbe"} err="failed to get container status \"86b26ad149c6d9ceed05efeafb60f9c4bbc9a36a69db957965614d23a6c4fcbe\": rpc error: code = NotFound desc = could not find container \"86b26ad149c6d9ceed05efeafb60f9c4bbc9a36a69db957965614d23a6c4fcbe\": container with ID starting with 86b26ad149c6d9ceed05efeafb60f9c4bbc9a36a69db957965614d23a6c4fcbe not found: ID does not exist"
Oct 30 22:23:21.568186 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:23:21.568163    4477 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7278-ea2049\" is forbidden: User \"system:node:ip-10-0-130-77.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-77.ec2.internal' and this object" podUID="ed1fed68-255c-4925-9d03-8e1b638539e6" pod="opendatahub/test-curl-7278-ea2049"
Oct 30 22:23:22.195502 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:22.195470    4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed1fed68-255c-4925-9d03-8e1b638539e6" path="/var/lib/kubelet/pods/ed1fed68-255c-4925-9d03-8e1b638539e6/volumes"
Oct 30 22:23:24.576124 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:24.576092    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ca259734-6d97f789c5-dfxt8"]
Oct 30 22:23:24.576915 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:24.576896    4477 cpu_manager.go:414] "containerMap: removing container" podUID="ed1fed68-255c-4925-9d03-8e1b638539e6" containerName="test-curl-7278-ea2049"
Oct 30 22:23:24.576969 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:24.576915    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed1fed68-255c-4925-9d03-8e1b638539e6" containerName="test-curl-7278-ea2049"
Oct 30 22:23:24.577170 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:24.577151    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed1fed68-255c-4925-9d03-8e1b638539e6" containerName="test-curl-7278-ea2049"
Oct 30 22:23:24.582033 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:24.582009    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ca259734-6d97f789c5-dfxt8"
Oct 30 22:23:24.597529 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:24.597509    4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-ca259734\""
Oct 30 22:23:24.608385 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:24.607902    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-ca259734-dockercfg-wrblz\""
Oct 30 22:23:24.613681 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:24.613656    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ca259734-6d97f789c5-dfxt8"]
Oct 30 22:23:24.645157 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:24.645128    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa7d15e2-9f74-48c2-b55f-73c2179b5e4f-plugins-config-volume\") pod \"payload-processing-e2e-ait-ca259734-6d97f789c5-dfxt8\" (UID: \"aa7d15e2-9f74-48c2-b55f-73c2179b5e4f\") " pod="openshift-ingress/payload-processing-e2e-ait-ca259734-6d97f789c5-dfxt8"
Oct 30 22:23:24.645265 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:24.645176    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nb2lk\" (UniqueName: \"kubernetes.io/projected/aa7d15e2-9f74-48c2-b55f-73c2179b5e4f-kube-api-access-nb2lk\") pod \"payload-processing-e2e-ait-ca259734-6d97f789c5-dfxt8\" (UID: \"aa7d15e2-9f74-48c2-b55f-73c2179b5e4f\") " pod="openshift-ingress/payload-processing-e2e-ait-ca259734-6d97f789c5-dfxt8"
Oct 30 22:23:24.652607 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:24.652573    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ca259734-86c66dfcf9-5dzzk"]
Oct 30 22:23:24.657711 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:24.657690    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ca259734-86c66dfcf9-5dzzk"
Oct 30 22:23:24.675566 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:24.675545    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ca259734-86c66dfcf9-5dzzk"]
Oct 30 22:23:24.762065 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:24.756991    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfk26\" (UniqueName: \"kubernetes.io/projected/6ba28fff-2bad-48f3-8d15-f4dc174ce26b-kube-api-access-tfk26\") pod \"payload-pre-processing-e2e-ait-ca259734-86c66dfcf9-5dzzk\" (UID: \"6ba28fff-2bad-48f3-8d15-f4dc174ce26b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ca259734-86c66dfcf9-5dzzk"
Oct 30 22:23:24.762065 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:24.757272    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa7d15e2-9f74-48c2-b55f-73c2179b5e4f-plugins-config-volume\") pod \"payload-processing-e2e-ait-ca259734-6d97f789c5-dfxt8\" (UID: \"aa7d15e2-9f74-48c2-b55f-73c2179b5e4f\") " pod="openshift-ingress/payload-processing-e2e-ait-ca259734-6d97f789c5-dfxt8"
Oct 30 22:23:24.762065 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:24.757310    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nb2lk\" (UniqueName: \"kubernetes.io/projected/aa7d15e2-9f74-48c2-b55f-73c2179b5e4f-kube-api-access-nb2lk\") pod \"payload-processing-e2e-ait-ca259734-6d97f789c5-dfxt8\" (UID: \"aa7d15e2-9f74-48c2-b55f-73c2179b5e4f\") " pod="openshift-ingress/payload-processing-e2e-ait-ca259734-6d97f789c5-dfxt8"
Oct 30 22:23:24.762065 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:24.757359    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ba28fff-2bad-48f3-8d15-f4dc174ce26b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ca259734-86c66dfcf9-5dzzk\" (UID: \"6ba28fff-2bad-48f3-8d15-f4dc174ce26b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ca259734-86c66dfcf9-5dzzk"
Oct 30 22:23:24.762065 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:24.758393    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa7d15e2-9f74-48c2-b55f-73c2179b5e4f-plugins-config-volume\") pod \"payload-processing-e2e-ait-ca259734-6d97f789c5-dfxt8\" (UID: \"aa7d15e2-9f74-48c2-b55f-73c2179b5e4f\") " pod="openshift-ingress/payload-processing-e2e-ait-ca259734-6d97f789c5-dfxt8"
Oct 30 22:23:24.801145 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:24.801111    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nb2lk\" (UniqueName: \"kubernetes.io/projected/aa7d15e2-9f74-48c2-b55f-73c2179b5e4f-kube-api-access-nb2lk\") pod \"payload-processing-e2e-ait-ca259734-6d97f789c5-dfxt8\" (UID: \"aa7d15e2-9f74-48c2-b55f-73c2179b5e4f\") " pod="openshift-ingress/payload-processing-e2e-ait-ca259734-6d97f789c5-dfxt8"
Oct 30 22:23:24.859548 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:24.859449    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ba28fff-2bad-48f3-8d15-f4dc174ce26b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ca259734-86c66dfcf9-5dzzk\" (UID: \"6ba28fff-2bad-48f3-8d15-f4dc174ce26b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ca259734-86c66dfcf9-5dzzk"
Oct 30 22:23:24.860586 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:24.860553    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfk26\" (UniqueName: \"kubernetes.io/projected/6ba28fff-2bad-48f3-8d15-f4dc174ce26b-kube-api-access-tfk26\") pod \"payload-pre-processing-e2e-ait-ca259734-86c66dfcf9-5dzzk\" (UID: \"6ba28fff-2bad-48f3-8d15-f4dc174ce26b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ca259734-86c66dfcf9-5dzzk"
Oct 30 22:23:24.862250 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:24.862224    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ba28fff-2bad-48f3-8d15-f4dc174ce26b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ca259734-86c66dfcf9-5dzzk\" (UID: \"6ba28fff-2bad-48f3-8d15-f4dc174ce26b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ca259734-86c66dfcf9-5dzzk"
Oct 30 22:23:24.881576 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:24.881541    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfk26\" (UniqueName: \"kubernetes.io/projected/6ba28fff-2bad-48f3-8d15-f4dc174ce26b-kube-api-access-tfk26\") pod \"payload-pre-processing-e2e-ait-ca259734-86c66dfcf9-5dzzk\" (UID: \"6ba28fff-2bad-48f3-8d15-f4dc174ce26b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ca259734-86c66dfcf9-5dzzk"
Oct 30 22:23:24.897456 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:24.897429    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ca259734-6d97f789c5-dfxt8"
Oct 30 22:23:24.973184 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:24.972508    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ca259734-86c66dfcf9-5dzzk"
Oct 30 22:23:25.247854 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:23:25.247808    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa7d15e2_9f74_48c2_b55f_73c2179b5e4f.slice/crio-c63e32364f0f5ead3a0403697e5c30a638633720a889d2f4922b10dcda646103 WatchSource:0}: Error finding container c63e32364f0f5ead3a0403697e5c30a638633720a889d2f4922b10dcda646103: Status 404 returned error can't find the container with id c63e32364f0f5ead3a0403697e5c30a638633720a889d2f4922b10dcda646103
Oct 30 22:23:25.255820 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:25.255796    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ca259734-6d97f789c5-dfxt8"]
Oct 30 22:23:25.332295 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:25.332264    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ca259734-86c66dfcf9-5dzzk"]
Oct 30 22:23:25.333615 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:23:25.333573    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ba28fff_2bad_48f3_8d15_f4dc174ce26b.slice/crio-82f73d36426fd5378281fae2dab7117ab4181f6af9a0241fd028f7a3c47f9d18 WatchSource:0}: Error finding container 82f73d36426fd5378281fae2dab7117ab4181f6af9a0241fd028f7a3c47f9d18: Status 404 returned error can't find the container with id 82f73d36426fd5378281fae2dab7117ab4181f6af9a0241fd028f7a3c47f9d18
Oct 30 22:23:25.573457 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:25.573366    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ca259734-6d97f789c5-dfxt8" event={"ID":"aa7d15e2-9f74-48c2-b55f-73c2179b5e4f","Type":"ContainerStarted","Data":"c63e32364f0f5ead3a0403697e5c30a638633720a889d2f4922b10dcda646103"}
Oct 30 22:23:25.574812 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:25.574781    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ca259734-86c66dfcf9-5dzzk" event={"ID":"6ba28fff-2bad-48f3-8d15-f4dc174ce26b","Type":"ContainerStarted","Data":"82f73d36426fd5378281fae2dab7117ab4181f6af9a0241fd028f7a3c47f9d18"}
Oct 30 22:23:26.275358 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:23:26.275320    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 22:23:28.611728 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:28.611689    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ca259734-6d97f789c5-dfxt8" event={"ID":"aa7d15e2-9f74-48c2-b55f-73c2179b5e4f","Type":"ContainerStarted","Data":"4f7b28839389acbe892db8ecf9b284c71324a063df4bacc0bdd3142b1cd9098e"}
Oct 30 22:23:28.612202 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:28.611859    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-ca259734-6d97f789c5-dfxt8"
Oct 30 22:23:28.613400 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:28.613375    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ca259734-86c66dfcf9-5dzzk" event={"ID":"6ba28fff-2bad-48f3-8d15-f4dc174ce26b","Type":"ContainerStarted","Data":"4bd3aca9dbc3f00fddf33a1296d850d768e89e3590dbb1631789cbab4340281e"}
Oct 30 22:23:28.613514 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:28.613443    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-ca259734-86c66dfcf9-5dzzk"
Oct 30 22:23:28.655348 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:28.655302    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-ca259734-6d97f789c5-dfxt8" podStartSLOduration=1.809387481 podStartE2EDuration="4.655290518s" totalImagesPullingTime="2.845903037s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:23:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:23:28.652273211 +0000 UTC m=+992.578199416" watchObservedRunningTime="2026-10-30 22:23:28.655290518 +0000 UTC m=+992.581216725"
Oct 30 22:23:28.700193 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:28.700052    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-ca259734-86c66dfcf9-5dzzk" podStartSLOduration=1.9876786339999999 podStartE2EDuration="4.700021608s" totalImagesPullingTime="2.712342974s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:23:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:23:28.694992641 +0000 UTC m=+992.620918846" watchObservedRunningTime="2026-10-30 22:23:28.700021608 +0000 UTC m=+992.625947814"
Oct 30 22:23:33.974393 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:33.973195    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0ad7c5/premium-w4-0ad7c5-kserve-7bbf78cd4-pm6wc" podUID="5c10371e-56e7-42ec-9ef8-0802e15cb2ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:34.087572 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:34.087531    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/distinct-w2-421583-kserve-69cf5f4d6b-qbgnt" podUID="510ad10d-0554-47c8-9b55-deb359795071" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Oct 30 22:23:34.266218 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:34.266122    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/distinct-w3-d52c59-kserve-ffb5f5b96-l58xl" podUID="c7b1483b-2631-4aa2-a888-3096761d5494" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Oct 30 22:23:34.293167 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:34.293121    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/premium-w5-54114e-kserve-57f6747bcc-7wjrw" podUID="61990d22-ab8f-45bc-a4d6-4c8d528742f1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:34.415216 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:34.415170    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-2-w4-0ad7c5-kserve-77c5467fcb-skspk" podUID="430bd879-2a89-4070-bb1b-9b9958a261b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:34.442285 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:34.442245    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/premium-w6-012ba0-kserve-658654dbdb-d2k27" podUID="085847e5-4f50-419e-984f-b612b104ed90" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:34.582067 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:34.581955    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/distinct-2-w0-09d0d5-kserve-7dcf89b98b-7cjph" podUID="8cbd04b4-bb3e-4bbb-a41a-46131c9de051" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded"
Oct 30 22:23:34.927065 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:34.926158    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/simulator-w1-2b1e88-kserve-6c579b99f8-q5wpq" podUID="722f10d8-b7a2-4bf7-a274-9669529ef427" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:34.976180 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:34.976131    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0ad7c5/premium-w4-0ad7c5-kserve-7bbf78cd4-pm6wc" podUID="5c10371e-56e7-42ec-9ef8-0802e15cb2ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:34.977521 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:34.977496    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0ad7c5/premium-w4-0ad7c5-kserve-7bbf78cd4-pm6wc"
Oct 30 22:23:35.086392 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:35.086057    4477 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-2b1e88/simulator-w1-2b1e88-kserve-6c579b99f8-q5wpq" podUID="722f10d8-b7a2-4bf7-a274-9669529ef427" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:35.092377 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:35.092337    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/distinct-w2-421583-kserve-69cf5f4d6b-qbgnt" podUID="510ad10d-0554-47c8-9b55-deb359795071" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:35.092941 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:35.092921    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-421583/distinct-w2-421583-kserve-69cf5f4d6b-qbgnt"
Oct 30 22:23:35.360058 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:35.303137    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/distinct-2-w6-012ba0-kserve-78dbcd8c46-hqdfm" podUID="8326b014-989b-4722-befb-6fa8ea60688b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:35.360058 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:35.331923    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/distinct-w3-d52c59-kserve-ffb5f5b96-l58xl" podUID="c7b1483b-2631-4aa2-a888-3096761d5494" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Oct 30 22:23:35.360058 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:35.332338    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d52c59/distinct-w3-d52c59-kserve-ffb5f5b96-l58xl"
Oct 30 22:23:35.360058 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:35.358766    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/premium-w5-54114e-kserve-57f6747bcc-7wjrw" podUID="61990d22-ab8f-45bc-a4d6-4c8d528742f1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded"
Oct 30 22:23:35.360058 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:35.359086    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-54114e/premium-w5-54114e-kserve-57f6747bcc-7wjrw"
Oct 30 22:23:35.437587 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:35.435139    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-2-w4-0ad7c5-kserve-77c5467fcb-skspk" podUID="430bd879-2a89-4070-bb1b-9b9958a261b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:35.437587 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:35.436323    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-2-w4-0ad7c5-kserve-77c5467fcb-skspk"
Oct 30 22:23:35.437587 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:35.437471    4477 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-09d0d5/distinct-2-w0-09d0d5-kserve-7dcf89b98b-7cjph" podUID="8cbd04b4-bb3e-4bbb-a41a-46131c9de051" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:35.479106 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:35.451212    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/premium-w6-012ba0-kserve-658654dbdb-d2k27" podUID="085847e5-4f50-419e-984f-b612b104ed90" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:35.479106 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:35.452352    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-012ba0/premium-w6-012ba0-kserve-658654dbdb-d2k27"
Oct 30 22:23:35.676310 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:35.600900    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/distinct-2-w0-09d0d5-kserve-7dcf89b98b-7cjph" podUID="8cbd04b4-bb3e-4bbb-a41a-46131c9de051" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:35.676310 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:35.660714    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-09d0d5/distinct-2-w0-09d0d5-kserve-7dcf89b98b-7cjph"
Oct 30 22:23:35.973121 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:35.946122    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/simulator-w1-2b1e88-kserve-6c579b99f8-q5wpq" podUID="722f10d8-b7a2-4bf7-a274-9669529ef427" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:35.973121 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:35.948496    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-2b1e88/simulator-w1-2b1e88-kserve-6c579b99f8-q5wpq"
Oct 30 22:23:36.010828 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:36.010785    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0ad7c5/premium-w4-0ad7c5-kserve-7bbf78cd4-pm6wc" podUID="5c10371e-56e7-42ec-9ef8-0802e15cb2ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:36.165067 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:36.108708    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/distinct-w2-421583-kserve-69cf5f4d6b-qbgnt" podUID="510ad10d-0554-47c8-9b55-deb359795071" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:36.315293 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:36.315197    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/distinct-2-w6-012ba0-kserve-78dbcd8c46-hqdfm" podUID="8326b014-989b-4722-befb-6fa8ea60688b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:36.332408 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:36.332367    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/distinct-w3-d52c59-kserve-ffb5f5b96-l58xl" podUID="c7b1483b-2631-4aa2-a888-3096761d5494" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:36.414093 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:36.392329    4477 patch_prober.go:29] interesting pod/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-crp67 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:23:36.414093 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:36.412068    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-crp67" podUID="2f5f8741-f496-46a2-b357-72c0e27ea2ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:36.414093 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:36.395834    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/premium-w5-54114e-kserve-57f6747bcc-7wjrw" podUID="61990d22-ab8f-45bc-a4d6-4c8d528742f1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded"
Oct 30 22:23:36.456183 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:36.456139    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-2-w4-0ad7c5-kserve-77c5467fcb-skspk" podUID="430bd879-2a89-4070-bb1b-9b9958a261b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:36.494158 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:36.479316    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/premium-w6-012ba0-kserve-658654dbdb-d2k27" podUID="085847e5-4f50-419e-984f-b612b104ed90" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:36.495357 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:36.495328    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-2b1e88/simulator-w1-2b1e88-kserve-6c579b99f8-q5wpq"
Oct 30 22:23:36.496517 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:36.496500    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-012ba0/distinct-2-w6-012ba0-kserve-78dbcd8c46-hqdfm"
Oct 30 22:23:36.601837 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:36.601747    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/distinct-2-w0-09d0d5-kserve-7dcf89b98b-7cjph" podUID="8cbd04b4-bb3e-4bbb-a41a-46131c9de051" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:36.797363 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:36.797324    4477 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-012ba0/distinct-2-w6-012ba0-kserve-78dbcd8c46-hqdfm" podUID="8326b014-989b-4722-befb-6fa8ea60688b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:37.017371 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:37.017330    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0ad7c5/premium-w4-0ad7c5-kserve-7bbf78cd4-pm6wc" podUID="5c10371e-56e7-42ec-9ef8-0802e15cb2ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:37.208913 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:37.208854    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/distinct-w2-421583-kserve-69cf5f4d6b-qbgnt" podUID="510ad10d-0554-47c8-9b55-deb359795071" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:37.273260 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:37.273156    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/simulator-w1-2b1e88-kserve-6c579b99f8-q5wpq" podUID="722f10d8-b7a2-4bf7-a274-9669529ef427" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:37.320305 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:37.320247    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/distinct-2-w6-012ba0-kserve-78dbcd8c46-hqdfm" podUID="8326b014-989b-4722-befb-6fa8ea60688b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:37.420396 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:37.418155    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/premium-w5-54114e-kserve-57f6747bcc-7wjrw" podUID="61990d22-ab8f-45bc-a4d6-4c8d528742f1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:37.420396 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:37.418559    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/distinct-w3-d52c59-kserve-ffb5f5b96-l58xl" podUID="c7b1483b-2631-4aa2-a888-3096761d5494" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Oct 30 22:23:37.463626 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:37.463511    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-2-w4-0ad7c5-kserve-77c5467fcb-skspk" podUID="430bd879-2a89-4070-bb1b-9b9958a261b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:37.493374 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:37.493332    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/premium-w6-012ba0-kserve-658654dbdb-d2k27" podUID="085847e5-4f50-419e-984f-b612b104ed90" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:37.507342 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:37.507314    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-09d0d5/distinct-2-w0-09d0d5-kserve-7dcf89b98b-7cjph"
Oct 30 22:23:38.025179 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:38.025138    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0ad7c5/premium-w4-0ad7c5-kserve-7bbf78cd4-pm6wc" podUID="5c10371e-56e7-42ec-9ef8-0802e15cb2ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:38.262585 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:38.262535    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/distinct-w2-421583-kserve-69cf5f4d6b-qbgnt" podUID="510ad10d-0554-47c8-9b55-deb359795071" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:38.286789 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:38.286670    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/simulator-w1-2b1e88-kserve-6c579b99f8-q5wpq" podUID="722f10d8-b7a2-4bf7-a274-9669529ef427" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": context deadline exceeded"
Oct 30 22:23:38.323205 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:38.323161    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/distinct-2-w6-012ba0-kserve-78dbcd8c46-hqdfm" podUID="8326b014-989b-4722-befb-6fa8ea60688b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Oct 30 22:23:38.545119 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:38.523793    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/premium-w5-54114e-kserve-57f6747bcc-7wjrw" podUID="61990d22-ab8f-45bc-a4d6-4c8d528742f1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": context deadline exceeded"
Oct 30 22:23:38.545119 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:38.524151    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d52c59/distinct-w3-d52c59-kserve-ffb5f5b96-l58xl" podUID="c7b1483b-2631-4aa2-a888-3096761d5494" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Oct 30 22:23:38.551721 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:38.547091    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-2-w4-0ad7c5-kserve-77c5467fcb-skspk" podUID="430bd879-2a89-4070-bb1b-9b9958a261b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:38.564107 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:38.558387    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/premium-w6-012ba0-kserve-658654dbdb-d2k27" podUID="085847e5-4f50-419e-984f-b612b104ed90" containerName="main" probeResult="failure" output="Get \"https://10.134.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:39.029220 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:39.029175    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0ad7c5/premium-w4-0ad7c5-kserve-7bbf78cd4-pm6wc" podUID="5c10371e-56e7-42ec-9ef8-0802e15cb2ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:39.151789 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:39.151758    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-2b1e88/simulator-w1-2b1e88-kserve-6c579b99f8-q5wpq"
Oct 30 22:23:39.152192 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:39.152175    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-421583/distinct-w2-421583-kserve-69cf5f4d6b-qbgnt"
Oct 30 22:23:39.152405 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:39.152394    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-012ba0/premium-w6-012ba0-kserve-658654dbdb-d2k27"
Oct 30 22:23:39.349645 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:39.288222    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/simulator-w1-2b1e88-kserve-6c579b99f8-q5wpq" podUID="722f10d8-b7a2-4bf7-a274-9669529ef427" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:39.353512 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:39.353485    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-6wn2g"]
Oct 30 22:23:39.382433 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:39.355290    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/distinct-2-w6-012ba0-kserve-78dbcd8c46-hqdfm" podUID="8326b014-989b-4722-befb-6fa8ea60688b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:39.392959 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:39.392932    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-6wn2g"]
Oct 30 22:23:39.393109 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:39.393055    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-6wn2g"
Oct 30 22:23:39.393690 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:39.393671    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d52c59/distinct-w3-d52c59-kserve-ffb5f5b96-l58xl"
Oct 30 22:23:39.442136 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:39.441315    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-n6fb8"]
Oct 30 22:23:39.448025 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:39.448002    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0ad7c5/premium-w4-0ad7c5-kserve-7bbf78cd4-pm6wc"
Oct 30 22:23:39.448238 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:39.448223    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-2b1e88/simulator-w1-2b1e88-kserve-6c579b99f8-q5wpq"
Oct 30 22:23:39.448462 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:39.448448    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-n6fb8"
Oct 30 22:23:39.449973 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:39.449952    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-012ba0/distinct-2-w6-012ba0-kserve-78dbcd8c46-hqdfm"
Oct 30 22:23:39.463066 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:39.463036    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-n6fb8"]
Oct 30 22:23:39.489724 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:39.489694    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-2-w4-0ad7c5-kserve-77c5467fcb-skspk"
Oct 30 22:23:39.499258 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:39.499229    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96dc25ab-312f-4915-80ad-35c864d7dfec-credential-socket\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-n6fb8\" (UID: \"96dc25ab-312f-4915-80ad-35c864d7dfec\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-n6fb8"
Oct 30 22:23:39.499462 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:39.499439    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96dc25ab-312f-4915-80ad-35c864d7dfec-istio-envoy\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-n6fb8\" (UID: \"96dc25ab-312f-4915-80ad-35c864d7dfec\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-n6fb8"
Oct 30 22:23:39.499607 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:39.499589    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96dc25ab-312f-4915-80ad-35c864d7dfec-workload-certs\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-n6fb8\" (UID: \"96dc25ab-312f-4915-80ad-35c864d7dfec\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-n6fb8"
Oct 30 22:23:39.499798 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:39.499781    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96dc25ab-312f-4915-80ad-35c864d7dfec-istio-data\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-n6fb8\" (UID: \"96dc25ab-312f-4915-80ad-35c864d7dfec\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-n6fb8"
Oct 30 22:23:39.499932 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:39.499911    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/720aa32b-9fc1-4013-bd84-a751a9f88fca-istiod-ca-cert\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-6wn2g\" (UID: \"720aa32b-9fc1-4013-bd84-a751a9f88fca\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-6wn2g"
Oct 30 22:23:39.500153 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:39.500131    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpqdf\" (UniqueName: \"kubernetes.io/projected/720aa32b-9fc1-4013-bd84-a751a9f88fca-kube-api-access-gpqdf\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-6wn2g\" (UID: \"720aa32b-9fc1-4013-bd84-a751a9f88fca\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-6wn2g"
Oct 30 22:23:39.500307 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:39.500288    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/720aa32b-9fc1-4013-bd84-a751a9f88fca-istio-token\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-6wn2g\" (UID: \"720aa32b-9fc1-4013-bd84-a751a9f88fca\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-6wn2g"
Oct 30 22:23:39.500463 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:39.500444    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/720aa32b-9fc1-4013-bd84-a751a9f88fca-istio-podinfo\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-6wn2g\" (UID: \"720aa32b-9fc1-4013-bd84-a751a9f88fca\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-6wn2g"
Oct 30 22:23:39.500625 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:39.500606    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/720aa32b-9fc1-4013-bd84-a751a9f88fca-istio-envoy\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-6wn2g\" (UID: \"720aa32b-9fc1-4013-bd84-a751a9f88fca\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-6wn2g"
Oct 30 22:23:39.500750 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:39.500734    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/720aa32b-9fc1-4013-bd84-a751a9f88fca-workload-socket\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-6wn2g\" (UID: \"720aa32b-9fc1-4013-bd84-a751a9f88fca\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-6wn2g"
Oct 30 22:23:39.501645 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:39.501625    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96dc25ab-312f-4915-80ad-35c864d7dfec-istio-podinfo\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-n6fb8\" (UID: \"96dc25ab-312f-4915-80ad-35c864d7dfec\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-n6fb8"
Oct 30 22:23:39.501999 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:39.501978    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96dc25ab-312f-4915-80ad-35c864d7dfec-istio-token\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-n6fb8\" (UID: \"96dc25ab-312f-4915-80ad-35c864d7dfec\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-n6fb8"
Oct 30 22:23:39.502174 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:39.502156    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlrn5\" (UniqueName: \"kubernetes.io/projected/96dc25ab-312f-4915-80ad-35c864d7dfec-kube-api-access-nlrn5\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-n6fb8\" (UID: \"96dc25ab-312f-4915-80ad-35c864d7dfec\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-n6fb8"
Oct 30 22:23:39.502334 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:39.502315    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96dc25ab-312f-4915-80ad-35c864d7dfec-istiod-ca-cert\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-n6fb8\" (UID: \"96dc25ab-312f-4915-80ad-35c864d7dfec\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-n6fb8"
Oct 30 22:23:39.502458 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:39.502441    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/720aa32b-9fc1-4013-bd84-a751a9f88fca-workload-certs\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-6wn2g\" (UID: \"720aa32b-9fc1-4013-bd84-a751a9f88fca\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-6wn2g"
Oct 30 22:23:39.502639 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:39.502620    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/720aa32b-9fc1-4013-bd84-a751a9f88fca-istio-data\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-6wn2g\" (UID: \"720aa32b-9fc1-4013-bd84-a751a9f88fca\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-6wn2g"
Oct 30 22:23:39.502845 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:39.502824    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96dc25ab-312f-4915-80ad-35c864d7dfec-workload-socket\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-n6fb8\" (UID: \"96dc25ab-312f-4915-80ad-35c864d7dfec\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-n6fb8"
Oct 30 22:23:39.503102 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:39.503083    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/720aa32b-9fc1-4013-bd84-a751a9f88fca-credential-socket\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-6wn2g\" (UID: \"720aa32b-9fc1-4013-bd84-a751a9f88fca\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-6wn2g"
Oct 30 22:23:39.525921 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:39.525887    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/premium-w5-54114e-kserve-57f6747bcc-7wjrw" podUID="61990d22-ab8f-45bc-a4d6-4c8d528742f1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:23:39.545339 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:39.545313    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-54114e/premium-w5-54114e-kserve-57f6747bcc-7wjrw"
Oct 30 22:23:39.605087 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:39.604996    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/720aa32b-9fc1-4013-bd84-a751a9f88fca-istio-podinfo\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-6wn2g\" (UID: \"720aa32b-9fc1-4013-bd84-a751a9f88fca\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-6wn2g"
Oct 30 22:23:39.605087 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:39.605073    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/720aa32b-9fc1-4013-bd84-a751a9f88fca-istio-envoy\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-6wn2g\" (UID: \"720aa32b-9fc1-4013-bd84-a751a9f88fca\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-6wn2g"
Oct 30 22:23:39.605309 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:39.605103    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/720aa32b-9fc1-4013-bd84-a751a9f88fca-workload-socket\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-6wn2g\" (UID: \"720aa32b-9fc1-4013-bd84-a751a9f88fca\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-6wn2g"
Oct 30 22:23:39.605309 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:39.605126    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96dc25ab-312f-4915-80ad-35c864d7dfec-istio-podinfo\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-n6fb8\" (UID: \"96dc25ab-312f-4915-80ad-35c864d7dfec\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-n6fb8"
Oct 30 22:23:39.605309 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:39.605249    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96dc25ab-312f-4915-80ad-35c864d7dfec-istio-token\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-n6fb8\" (UID: \"96dc25ab-312f-4915-80ad-35c864d7dfec\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-n6fb8"
Oct 30 22:23:39.605309 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:39.605282    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlrn5\" (UniqueName: \"kubernetes.io/projected/96dc25ab-312f-4915-80ad-35c864d7dfec-kube-api-access-nlrn5\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-n6fb8\" (UID: \"96dc25ab-312f-4915-80ad-35c864d7dfec\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-n6fb8"
Oct 30 22:23:39.605529 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:39.605345    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96dc25ab-312f-4915-80ad-35c864d7dfec-istiod-ca-cert\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-n6fb8\" (UID: \"96dc25ab-312f-4915-80ad-35c864d7dfec\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-n6fb8"
Oct 30 22:23:39.605529 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:39.605378    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/720aa32b-9fc1-4013-bd84-a751a9f88fca-workload-certs\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-6wn2g\" (UID: \"720aa32b-9fc1-4013-bd84-a751a9f88fca\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-6wn2g"
Oct 30 22:23:39.605529 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:39.605425    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/720aa32b-9fc1-4013-bd84-a751a9f88fca-istio-data\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-6wn2g\" (UID: \"720aa32b-9fc1-4013-bd84-a751a9f88fca\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-6wn2g"
Oct 30 22:23:39.605529 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:39.605467    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96dc25ab-312f-4915-80ad-35c864d7dfec-workload-socket\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-n6fb8\" (UID: \"96dc25ab-312f-4915-80ad-35c864d7dfec\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-n6fb8"
Oct 30 22:23:39.605529 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:39.605498    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/720aa32b-9fc1-4013-bd84-a751a9f88fca-credential-socket\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-6wn2g\" (UID: \"720aa32b-9fc1-4013-bd84-a751a9f88fca\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-6wn2g"
Oct 30 22:23:39.605791 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:39.605552    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/720aa32b-9fc1-4013-bd84-a751a9f88fca-workload-socket\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-6wn2g\" (UID: \"720aa32b-9fc1-4013-bd84-a751a9f88fca\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-6wn2g"
Oct 30 22:23:39.605791 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:39.605599    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96dc25ab-312f-4915-80ad-35c864d7dfec-credential-socket\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-n6fb8\" (UID: \"96dc25ab-312f-4915-80ad-35c864d7dfec\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-n6fb8"
Oct 30 22:23:39.605791 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:39.605648    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96dc25ab-312f-4915-80ad-35c864d7dfec-istio-envoy\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-n6fb8\" (UID: \"96dc25ab-312f-4915-80ad-35c864d7dfec\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-n6fb8"
Oct 30 22:23:39.605791 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:39.605676    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96dc25ab-312f-4915-80ad-35c864d7dfec-workload-certs\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-n6fb8\" (UID: \"96dc25ab-312f-4915-80ad-35c864d7dfec\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-n6fb8"
Oct 30 22:23:39.605791 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:39.605720    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96dc25ab-312f-4915-80ad-35c864d7dfec-istio-data\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-n6fb8\" (UID: \"96dc25ab-312f-4915-80ad-35c864d7dfec\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-n6fb8"
Oct 30 22:23:39.605791 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:39.605751    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/720aa32b-9fc1-4013-bd84-a751a9f88fca-istiod-ca-cert\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-6wn2g\" (UID: \"720aa32b-9fc1-4013-bd84-a751a9f88fca\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-6wn2g"
Oct 30 22:23:39.606118 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:39.605789    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/720aa32b-9fc1-4013-bd84-a751a9f88fca-workload-certs\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-6wn2g\" (UID: \"720aa32b-9fc1-4013-bd84-a751a9f88fca\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-6wn2g"
Oct 30 22:23:39.606118 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:39.605801    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpqdf\" (UniqueName: \"kubernetes.io/projected/720aa32b-9fc1-4013-bd84-a751a9f88fca-kube-api-access-gpqdf\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-6wn2g\" (UID: \"720aa32b-9fc1-4013-bd84-a751a9f88fca\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-6wn2g"
Oct 30 22:23:39.606118 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:39.605826    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/720aa32b-9fc1-4013-bd84-a751a9f88fca-credential-socket\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-6wn2g\" (UID: \"720aa32b-9fc1-4013-bd84-a751a9f88fca\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-6wn2g"
Oct 30 22:23:39.606118 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:39.605856    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/720aa32b-9fc1-4013-bd84-a751a9f88fca-istio-token\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-6wn2g\" (UID: \"720aa32b-9fc1-4013-bd84-a751a9f88fca\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-6wn2g"
Oct 30 22:23:39.606118 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:39.606077    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96dc25ab-312f-4915-80ad-35c864d7dfec-istiod-ca-cert\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-n6fb8\" (UID: \"96dc25ab-312f-4915-80ad-35c864d7dfec\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-n6fb8"
Oct 30 22:23:39.606368 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:39.606344    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/720aa32b-9fc1-4013-bd84-a751a9f88fca-istio-data\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-6wn2g\" (UID: \"720aa32b-9fc1-4013-bd84-a751a9f88fca\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-6wn2g"
Oct 30 22:23:39.606425 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:39.606410    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96dc25ab-312f-4915-80ad-35c864d7dfec-istio-data\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-n6fb8\" (UID: \"96dc25ab-312f-4915-80ad-35c864d7dfec\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-n6fb8"
Oct 30 22:23:39.606528 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:39.606506    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96dc25ab-312f-4915-80ad-35c864d7dfec-workload-socket\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-n6fb8\" (UID: \"96dc25ab-312f-4915-80ad-35c864d7dfec\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-n6fb8"
Oct 30 22:23:39.606673 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:39.606653    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96dc25ab-312f-4915-80ad-35c864d7dfec-credential-socket\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-n6fb8\" (UID: \"96dc25ab-312f-4915-80ad-35c864d7dfec\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-n6fb8"
Oct 30 22:23:39.606749 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:39.606674    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96dc25ab-312f-4915-80ad-35c864d7dfec-workload-certs\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-n6fb8\" (UID: \"96dc25ab-312f-4915-80ad-35c864d7dfec\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-n6fb8"
Oct 30 22:23:39.607212 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:39.607159    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/720aa32b-9fc1-4013-bd84-a751a9f88fca-istiod-ca-cert\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-6wn2g\" (UID: \"720aa32b-9fc1-4013-bd84-a751a9f88fca\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-6wn2g"
Oct 30 22:23:39.608417 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:39.608391    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96dc25ab-312f-4915-80ad-35c864d7dfec-istio-envoy\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-n6fb8\" (UID: \"96dc25ab-312f-4915-80ad-35c864d7dfec\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-n6fb8"
Oct 30 22:23:39.608784 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:39.608757    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/720aa32b-9fc1-4013-bd84-a751a9f88fca-istio-podinfo\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-6wn2g\" (UID: \"720aa32b-9fc1-4013-bd84-a751a9f88fca\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-6wn2g"
Oct 30 22:23:39.609236 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:39.609211    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/720aa32b-9fc1-4013-bd84-a751a9f88fca-istio-envoy\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-6wn2g\" (UID: \"720aa32b-9fc1-4013-bd84-a751a9f88fca\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-6wn2g"
Oct 30 22:23:39.610718 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:39.610698    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96dc25ab-312f-4915-80ad-35c864d7dfec-istio-podinfo\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-n6fb8\" (UID: \"96dc25ab-312f-4915-80ad-35c864d7dfec\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-n6fb8"
Oct 30 22:23:39.624085 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:39.623769    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-ca259734-6d97f789c5-dfxt8"
Oct 30 22:23:39.624880 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:39.624800    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-ca259734-86c66dfcf9-5dzzk"
Oct 30 22:23:39.624880 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:39.624829    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpqdf\" (UniqueName: \"kubernetes.io/projected/720aa32b-9fc1-4013-bd84-a751a9f88fca-kube-api-access-gpqdf\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-6wn2g\" (UID: \"720aa32b-9fc1-4013-bd84-a751a9f88fca\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-6wn2g"
Oct 30 22:23:39.625189 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:39.625163    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlrn5\" (UniqueName: \"kubernetes.io/projected/96dc25ab-312f-4915-80ad-35c864d7dfec-kube-api-access-nlrn5\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-n6fb8\" (UID: \"96dc25ab-312f-4915-80ad-35c864d7dfec\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-n6fb8"
Oct 30 22:23:39.626454 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:39.626435    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/720aa32b-9fc1-4013-bd84-a751a9f88fca-istio-token\") pod \"e2e-worker-w5-54114e-openshift-default-7fc467ddd7-6wn2g\" (UID: \"720aa32b-9fc1-4013-bd84-a751a9f88fca\") " pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-6wn2g"
Oct 30 22:23:39.626649 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:39.626631    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96dc25ab-312f-4915-80ad-35c864d7dfec-istio-token\") pod \"e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-n6fb8\" (UID: \"96dc25ab-312f-4915-80ad-35c864d7dfec\") " pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-n6fb8"
Oct 30 22:23:39.747588 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:39.747554    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-6wn2g"
Oct 30 22:23:39.771489 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:39.771464    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-n6fb8"
Oct 30 22:23:40.076453 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:40.076429    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-6wn2g"]
Oct 30 22:23:40.078374 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:23:40.078347    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod720aa32b_9fc1_4013_bd84_a751a9f88fca.slice/crio-942c3e06f8c22f4707ea56c1e3e2573f0ef5a0721fb3bb296e7d8b108139b489 WatchSource:0}: Error finding container 942c3e06f8c22f4707ea56c1e3e2573f0ef5a0721fb3bb296e7d8b108139b489: Status 404 returned error can't find the container with id 942c3e06f8c22f4707ea56c1e3e2573f0ef5a0721fb3bb296e7d8b108139b489
Oct 30 22:23:40.080880 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:40.080843    4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:23:40.080945 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:40.080912    4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:23:40.080981 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:40.080942    4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:23:40.097456 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:40.097435    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-n6fb8"]
Oct 30 22:23:40.098342 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:23:40.098316    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96dc25ab_312f_4915_80ad_35c864d7dfec.slice/crio-91c1634190b110e4e6c4573370ac69d5f7d4daf0e69e4ab1758bd4d0f31804d2 WatchSource:0}: Error finding container 91c1634190b110e4e6c4573370ac69d5f7d4daf0e69e4ab1758bd4d0f31804d2: Status 404 returned error can't find the container with id 91c1634190b110e4e6c4573370ac69d5f7d4daf0e69e4ab1758bd4d0f31804d2
Oct 30 22:23:40.100741 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:40.100711    4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:23:40.100850 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:40.100770    4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:23:40.100850 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:40.100812    4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:23:41.031092 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:41.030989    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-6wn2g" event={"ID":"720aa32b-9fc1-4013-bd84-a751a9f88fca","Type":"ContainerStarted","Data":"94cd3a6c54b45c2a429d06dc44bb8f107fb0c0a343256f42c08632476e09b10b"}
Oct 30 22:23:41.032504 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:41.031316    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-6wn2g" event={"ID":"720aa32b-9fc1-4013-bd84-a751a9f88fca","Type":"ContainerStarted","Data":"942c3e06f8c22f4707ea56c1e3e2573f0ef5a0721fb3bb296e7d8b108139b489"}
Oct 30 22:23:41.035939 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:41.035414    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-n6fb8" event={"ID":"96dc25ab-312f-4915-80ad-35c864d7dfec","Type":"ContainerStarted","Data":"29833178f4731ebe85ef3a66f4fb3eda5ef0bdaad96caf106b833caebb7813ae"}
Oct 30 22:23:41.035939 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:41.035465    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-n6fb8" event={"ID":"96dc25ab-312f-4915-80ad-35c864d7dfec","Type":"ContainerStarted","Data":"91c1634190b110e4e6c4573370ac69d5f7d4daf0e69e4ab1758bd4d0f31804d2"}
Oct 30 22:23:41.104065 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:41.101021    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-6wn2g" podStartSLOduration=3.101003174 podStartE2EDuration="3.101003174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:23:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:23:41.088552542 +0000 UTC m=+1005.014478752" watchObservedRunningTime="2026-10-30 22:23:41.101003174 +0000 UTC m=+1005.026929381"
Oct 30 22:23:41.153438 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:41.153388    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-n6fb8" podStartSLOduration=3.153373377 podStartE2EDuration="3.153373377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:23:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:23:41.147416868 +0000 UTC m=+1005.073343073" watchObservedRunningTime="2026-10-30 22:23:41.153373377 +0000 UTC m=+1005.079299582"
Oct 30 22:23:41.748609 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:41.748580    4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-6wn2g"
Oct 30 22:23:41.750239 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:41.750213    4477 patch_prober.go:29] interesting pod/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-6wn2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 30 22:23:41.750393 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:41.750269    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-6wn2g" podUID="720aa32b-9fc1-4013-bd84-a751a9f88fca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 30 22:23:41.778302 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:41.774321    4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-n6fb8"
Oct 30 22:23:41.781426 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:41.778542    4477 patch_prober.go:29] interesting pod/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-n6fb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Oct 30 22:23:41.781659 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:41.781627    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-n6fb8" podUID="96dc25ab-312f-4915-80ad-35c864d7dfec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Oct 30 22:23:42.748767 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:42.748733    4477 patch_prober.go:29] interesting pod/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-6wn2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 30 22:23:42.749206 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:42.748787    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-6wn2g" podUID="720aa32b-9fc1-4013-bd84-a751a9f88fca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 30 22:23:42.773403 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:42.773372    4477 patch_prober.go:29] interesting pod/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-n6fb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Oct 30 22:23:42.773638 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:42.773613    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-n6fb8" podUID="96dc25ab-312f-4915-80ad-35c864d7dfec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Oct 30 22:23:43.748786 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:43.748749    4477 patch_prober.go:29] interesting pod/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-6wn2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 30 22:23:43.749214 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:43.748813    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-6wn2g" podUID="720aa32b-9fc1-4013-bd84-a751a9f88fca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 30 22:23:43.778068 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:43.774656    4477 patch_prober.go:29] interesting pod/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-n6fb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Oct 30 22:23:43.778068 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:43.774702    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-n6fb8" podUID="96dc25ab-312f-4915-80ad-35c864d7dfec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Oct 30 22:23:44.748081 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:44.748022    4477 patch_prober.go:29] interesting pod/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-6wn2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 30 22:23:44.748266 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:44.748114    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-6wn2g" podUID="720aa32b-9fc1-4013-bd84-a751a9f88fca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 30 22:23:44.753129 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:44.753103    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ca259734-7f54c968d7-v4pjb"]
Oct 30 22:23:44.761745 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:44.761726    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ca259734-7f54c968d7-v4pjb"
Oct 30 22:23:44.789246 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:44.789220    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-ca259734-serving-cert\""
Oct 30 22:23:44.812065 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:44.798523    4477 patch_prober.go:29] interesting pod/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-n6fb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Oct 30 22:23:44.812065 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:44.798583    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-n6fb8" podUID="96dc25ab-312f-4915-80ad-35c864d7dfec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Oct 30 22:23:44.837198 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:44.837159    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ca259734-7f54c968d7-v4pjb"]
Oct 30 22:23:44.940296 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:44.940254    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4b04c98-97ff-4c2c-bdf9-45d5e7e2006e-openshift-service-ca\") pod \"maas-api-e2e-ait-ca259734-7f54c968d7-v4pjb\" (UID: \"f4b04c98-97ff-4c2c-bdf9-45d5e7e2006e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ca259734-7f54c968d7-v4pjb"
Oct 30 22:23:44.940438 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:44.940373    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4b04c98-97ff-4c2c-bdf9-45d5e7e2006e-metrics-certs\") pod \"maas-api-e2e-ait-ca259734-7f54c968d7-v4pjb\" (UID: \"f4b04c98-97ff-4c2c-bdf9-45d5e7e2006e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ca259734-7f54c968d7-v4pjb"
Oct 30 22:23:44.940438 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:44.940401    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4b04c98-97ff-4c2c-bdf9-45d5e7e2006e-maas-api-tls\") pod \"maas-api-e2e-ait-ca259734-7f54c968d7-v4pjb\" (UID: \"f4b04c98-97ff-4c2c-bdf9-45d5e7e2006e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ca259734-7f54c968d7-v4pjb"
Oct 30 22:23:44.940754 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:44.940731    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctfzc\" (UniqueName: \"kubernetes.io/projected/f4b04c98-97ff-4c2c-bdf9-45d5e7e2006e-kube-api-access-ctfzc\") pod \"maas-api-e2e-ait-ca259734-7f54c968d7-v4pjb\" (UID: \"f4b04c98-97ff-4c2c-bdf9-45d5e7e2006e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ca259734-7f54c968d7-v4pjb"
Oct 30 22:23:45.042346 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:45.042266    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctfzc\" (UniqueName: \"kubernetes.io/projected/f4b04c98-97ff-4c2c-bdf9-45d5e7e2006e-kube-api-access-ctfzc\") pod \"maas-api-e2e-ait-ca259734-7f54c968d7-v4pjb\" (UID: \"f4b04c98-97ff-4c2c-bdf9-45d5e7e2006e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ca259734-7f54c968d7-v4pjb"
Oct 30 22:23:45.042497 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:45.042391    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4b04c98-97ff-4c2c-bdf9-45d5e7e2006e-openshift-service-ca\") pod \"maas-api-e2e-ait-ca259734-7f54c968d7-v4pjb\" (UID: \"f4b04c98-97ff-4c2c-bdf9-45d5e7e2006e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ca259734-7f54c968d7-v4pjb"
Oct 30 22:23:45.043031 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:45.043007    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4b04c98-97ff-4c2c-bdf9-45d5e7e2006e-openshift-service-ca\") pod \"maas-api-e2e-ait-ca259734-7f54c968d7-v4pjb\" (UID: \"f4b04c98-97ff-4c2c-bdf9-45d5e7e2006e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ca259734-7f54c968d7-v4pjb"
Oct 30 22:23:45.043174 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:45.043109    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4b04c98-97ff-4c2c-bdf9-45d5e7e2006e-metrics-certs\") pod \"maas-api-e2e-ait-ca259734-7f54c968d7-v4pjb\" (UID: \"f4b04c98-97ff-4c2c-bdf9-45d5e7e2006e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ca259734-7f54c968d7-v4pjb"
Oct 30 22:23:45.043174 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:45.043172    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4b04c98-97ff-4c2c-bdf9-45d5e7e2006e-maas-api-tls\") pod \"maas-api-e2e-ait-ca259734-7f54c968d7-v4pjb\" (UID: \"f4b04c98-97ff-4c2c-bdf9-45d5e7e2006e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ca259734-7f54c968d7-v4pjb"
Oct 30 22:23:45.043315 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:23:45.043301    4477 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-ca259734-serving-cert: secret "maas-api-e2e-ait-ca259734-serving-cert" not found
Oct 30 22:23:45.043371 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:23:45.043354    4477 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4b04c98-97ff-4c2c-bdf9-45d5e7e2006e-maas-api-tls podName:f4b04c98-97ff-4c2c-bdf9-45d5e7e2006e nodeName:}" failed. No retries permitted until 2026-10-30 22:23:45.543340335 +0000 UTC m=+1009.469266519 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f4b04c98-97ff-4c2c-bdf9-45d5e7e2006e-maas-api-tls") pod "maas-api-e2e-ait-ca259734-7f54c968d7-v4pjb" (UID: "f4b04c98-97ff-4c2c-bdf9-45d5e7e2006e") : secret "maas-api-e2e-ait-ca259734-serving-cert" not found
Oct 30 22:23:45.045739 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:45.045719    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4b04c98-97ff-4c2c-bdf9-45d5e7e2006e-metrics-certs\") pod \"maas-api-e2e-ait-ca259734-7f54c968d7-v4pjb\" (UID: \"f4b04c98-97ff-4c2c-bdf9-45d5e7e2006e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ca259734-7f54c968d7-v4pjb"
Oct 30 22:23:45.061745 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:45.061719    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctfzc\" (UniqueName: \"kubernetes.io/projected/f4b04c98-97ff-4c2c-bdf9-45d5e7e2006e-kube-api-access-ctfzc\") pod \"maas-api-e2e-ait-ca259734-7f54c968d7-v4pjb\" (UID: \"f4b04c98-97ff-4c2c-bdf9-45d5e7e2006e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ca259734-7f54c968d7-v4pjb"
Oct 30 22:23:45.552426 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:45.552391    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4b04c98-97ff-4c2c-bdf9-45d5e7e2006e-maas-api-tls\") pod \"maas-api-e2e-ait-ca259734-7f54c968d7-v4pjb\" (UID: \"f4b04c98-97ff-4c2c-bdf9-45d5e7e2006e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ca259734-7f54c968d7-v4pjb"
Oct 30 22:23:45.555019 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:45.554998    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4b04c98-97ff-4c2c-bdf9-45d5e7e2006e-maas-api-tls\") pod \"maas-api-e2e-ait-ca259734-7f54c968d7-v4pjb\" (UID: \"f4b04c98-97ff-4c2c-bdf9-45d5e7e2006e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ca259734-7f54c968d7-v4pjb"
Oct 30 22:23:45.745417 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:45.745380    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ca259734-7f54c968d7-v4pjb"
Oct 30 22:23:45.748852 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:45.748337    4477 patch_prober.go:29] interesting pod/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-6wn2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 30 22:23:45.748852 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:45.748395    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-6wn2g" podUID="720aa32b-9fc1-4013-bd84-a751a9f88fca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 30 22:23:45.776143 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:45.776094    4477 patch_prober.go:29] interesting pod/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-n6fb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Oct 30 22:23:45.776703 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:45.776672    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-n6fb8" podUID="96dc25ab-312f-4915-80ad-35c864d7dfec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Oct 30 22:23:46.423223 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:46.423185    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ca259734-7f54c968d7-v4pjb"]
Oct 30 22:23:46.748157 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:46.748118    4477 patch_prober.go:29] interesting pod/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-6wn2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body=
Oct 30 22:23:46.748363 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:46.748193    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-6wn2g" podUID="720aa32b-9fc1-4013-bd84-a751a9f88fca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused"
Oct 30 22:23:46.753702 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:46.753670    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1ba9ae08-999dc7c5d-wpffx"]
Oct 30 22:23:46.759252 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:46.759201    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1ba9ae08-999dc7c5d-wpffx"
Oct 30 22:23:46.772266 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:46.772239    4477 patch_prober.go:29] interesting pod/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-n6fb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Oct 30 22:23:46.772351 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:46.772286    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-n6fb8" podUID="96dc25ab-312f-4915-80ad-35c864d7dfec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Oct 30 22:23:46.778760 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:46.778669    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-1ba9ae08-serving-cert\""
Oct 30 22:23:46.784797 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:46.784775    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1ba9ae08-999dc7c5d-wpffx"]
Oct 30 22:23:46.926363 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:46.926313    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7969e73-ec10-41a1-a26f-6477c17e55fb-openshift-service-ca\") pod \"maas-api-e2e-ait-1ba9ae08-999dc7c5d-wpffx\" (UID: \"d7969e73-ec10-41a1-a26f-6477c17e55fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1ba9ae08-999dc7c5d-wpffx"
Oct 30 22:23:46.926564 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:46.926544    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7969e73-ec10-41a1-a26f-6477c17e55fb-metrics-certs\") pod \"maas-api-e2e-ait-1ba9ae08-999dc7c5d-wpffx\" (UID: \"d7969e73-ec10-41a1-a26f-6477c17e55fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1ba9ae08-999dc7c5d-wpffx"
Oct 30 22:23:46.926727 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:46.926707    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7969e73-ec10-41a1-a26f-6477c17e55fb-maas-api-tls\") pod \"maas-api-e2e-ait-1ba9ae08-999dc7c5d-wpffx\" (UID: \"d7969e73-ec10-41a1-a26f-6477c17e55fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1ba9ae08-999dc7c5d-wpffx"
Oct 30 22:23:46.926835 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:46.926818    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkbqp\" (UniqueName: \"kubernetes.io/projected/d7969e73-ec10-41a1-a26f-6477c17e55fb-kube-api-access-vkbqp\") pod \"maas-api-e2e-ait-1ba9ae08-999dc7c5d-wpffx\" (UID: \"d7969e73-ec10-41a1-a26f-6477c17e55fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1ba9ae08-999dc7c5d-wpffx"
Oct 30 22:23:46.934959 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:46.934922    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wx4cl"]
Oct 30 22:23:46.940504 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:46.940480    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wx4cl"
Oct 30 22:23:46.960587 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:46.960552    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wx4cl"]
Oct 30 22:23:46.963821 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:46.963794    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 30 22:23:46.964840 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:46.964797    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-ggg7r\""
Oct 30 22:23:46.965068 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:46.965034    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 30 22:23:46.965222 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:46.965207    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 30 22:23:47.048219 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:47.043209    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkbqp\" (UniqueName: \"kubernetes.io/projected/d7969e73-ec10-41a1-a26f-6477c17e55fb-kube-api-access-vkbqp\") pod \"maas-api-e2e-ait-1ba9ae08-999dc7c5d-wpffx\" (UID: \"d7969e73-ec10-41a1-a26f-6477c17e55fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1ba9ae08-999dc7c5d-wpffx"
Oct 30 22:23:47.048219 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:47.043328    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7969e73-ec10-41a1-a26f-6477c17e55fb-openshift-service-ca\") pod \"maas-api-e2e-ait-1ba9ae08-999dc7c5d-wpffx\" (UID: \"d7969e73-ec10-41a1-a26f-6477c17e55fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1ba9ae08-999dc7c5d-wpffx"
Oct 30 22:23:47.048219 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:47.043446    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gk9dp\" (UniqueName: \"kubernetes.io/projected/6f02c360-27f4-4a25-8155-436e260be559-kube-api-access-gk9dp\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wx4cl\" (UID: \"6f02c360-27f4-4a25-8155-436e260be559\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wx4cl"
Oct 30 22:23:47.048219 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:47.043485    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6f02c360-27f4-4a25-8155-436e260be559-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wx4cl\" (UID: \"6f02c360-27f4-4a25-8155-436e260be559\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wx4cl"
Oct 30 22:23:47.048219 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:47.043524    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7969e73-ec10-41a1-a26f-6477c17e55fb-metrics-certs\") pod \"maas-api-e2e-ait-1ba9ae08-999dc7c5d-wpffx\" (UID: \"d7969e73-ec10-41a1-a26f-6477c17e55fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1ba9ae08-999dc7c5d-wpffx"
Oct 30 22:23:47.048219 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:47.043563    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6f02c360-27f4-4a25-8155-436e260be559-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wx4cl\" (UID: \"6f02c360-27f4-4a25-8155-436e260be559\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wx4cl"
Oct 30 22:23:47.048219 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:47.043636    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f02c360-27f4-4a25-8155-436e260be559-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wx4cl\" (UID: \"6f02c360-27f4-4a25-8155-436e260be559\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wx4cl"
Oct 30 22:23:47.048219 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:47.043663    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f02c360-27f4-4a25-8155-436e260be559-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wx4cl\" (UID: \"6f02c360-27f4-4a25-8155-436e260be559\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wx4cl"
Oct 30 22:23:47.048219 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:47.043697    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6f02c360-27f4-4a25-8155-436e260be559-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wx4cl\" (UID: \"6f02c360-27f4-4a25-8155-436e260be559\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wx4cl"
Oct 30 22:23:47.048219 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:47.043733    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7969e73-ec10-41a1-a26f-6477c17e55fb-maas-api-tls\") pod \"maas-api-e2e-ait-1ba9ae08-999dc7c5d-wpffx\" (UID: \"d7969e73-ec10-41a1-a26f-6477c17e55fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1ba9ae08-999dc7c5d-wpffx"
Oct 30 22:23:47.048219 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:47.043777    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6f02c360-27f4-4a25-8155-436e260be559-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wx4cl\" (UID: \"6f02c360-27f4-4a25-8155-436e260be559\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wx4cl"
Oct 30 22:23:47.048219 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:47.045015    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7969e73-ec10-41a1-a26f-6477c17e55fb-openshift-service-ca\") pod \"maas-api-e2e-ait-1ba9ae08-999dc7c5d-wpffx\" (UID: \"d7969e73-ec10-41a1-a26f-6477c17e55fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1ba9ae08-999dc7c5d-wpffx"
Oct 30 22:23:47.048219 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:23:47.045358    4477 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-1ba9ae08-serving-cert: secret "maas-api-e2e-ait-1ba9ae08-serving-cert" not found
Oct 30 22:23:47.048219 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:23:47.045419    4477 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7969e73-ec10-41a1-a26f-6477c17e55fb-maas-api-tls podName:d7969e73-ec10-41a1-a26f-6477c17e55fb nodeName:}" failed. No retries permitted until 2026-10-30 22:23:47.545400976 +0000 UTC m=+1011.471327173 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d7969e73-ec10-41a1-a26f-6477c17e55fb-maas-api-tls") pod "maas-api-e2e-ait-1ba9ae08-999dc7c5d-wpffx" (UID: "d7969e73-ec10-41a1-a26f-6477c17e55fb") : secret "maas-api-e2e-ait-1ba9ae08-serving-cert" not found
Oct 30 22:23:47.053217 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:47.052494    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7969e73-ec10-41a1-a26f-6477c17e55fb-metrics-certs\") pod \"maas-api-e2e-ait-1ba9ae08-999dc7c5d-wpffx\" (UID: \"d7969e73-ec10-41a1-a26f-6477c17e55fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1ba9ae08-999dc7c5d-wpffx"
Oct 30 22:23:47.072782 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:47.072755    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkbqp\" (UniqueName: \"kubernetes.io/projected/d7969e73-ec10-41a1-a26f-6477c17e55fb-kube-api-access-vkbqp\") pod \"maas-api-e2e-ait-1ba9ae08-999dc7c5d-wpffx\" (UID: \"d7969e73-ec10-41a1-a26f-6477c17e55fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1ba9ae08-999dc7c5d-wpffx"
Oct 30 22:23:47.145011 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:47.144973    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f02c360-27f4-4a25-8155-436e260be559-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wx4cl\" (UID: \"6f02c360-27f4-4a25-8155-436e260be559\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wx4cl"
Oct 30 22:23:47.145201 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:47.145025    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f02c360-27f4-4a25-8155-436e260be559-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wx4cl\" (UID: \"6f02c360-27f4-4a25-8155-436e260be559\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wx4cl"
Oct 30 22:23:47.145201 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:47.145080    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6f02c360-27f4-4a25-8155-436e260be559-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wx4cl\" (UID: \"6f02c360-27f4-4a25-8155-436e260be559\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wx4cl"
Oct 30 22:23:47.145612 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:47.145175    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6f02c360-27f4-4a25-8155-436e260be559-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wx4cl\" (UID: \"6f02c360-27f4-4a25-8155-436e260be559\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wx4cl"
Oct 30 22:23:47.145956 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:47.145933    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gk9dp\" (UniqueName: \"kubernetes.io/projected/6f02c360-27f4-4a25-8155-436e260be559-kube-api-access-gk9dp\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wx4cl\" (UID: \"6f02c360-27f4-4a25-8155-436e260be559\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wx4cl"
Oct 30 22:23:47.146107 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:47.145986    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6f02c360-27f4-4a25-8155-436e260be559-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wx4cl\" (UID: \"6f02c360-27f4-4a25-8155-436e260be559\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wx4cl"
Oct 30 22:23:47.146107 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:47.146063    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6f02c360-27f4-4a25-8155-436e260be559-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wx4cl\" (UID: \"6f02c360-27f4-4a25-8155-436e260be559\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wx4cl"
Oct 30 22:23:47.147375 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:47.147345    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6f02c360-27f4-4a25-8155-436e260be559-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wx4cl\" (UID: \"6f02c360-27f4-4a25-8155-436e260be559\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wx4cl"
Oct 30 22:23:47.147780 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:47.147709    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6f02c360-27f4-4a25-8155-436e260be559-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wx4cl\" (UID: \"6f02c360-27f4-4a25-8155-436e260be559\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wx4cl"
Oct 30 22:23:47.147780 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:47.147733    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f02c360-27f4-4a25-8155-436e260be559-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wx4cl\" (UID: \"6f02c360-27f4-4a25-8155-436e260be559\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wx4cl"
Oct 30 22:23:47.148833 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:47.148803    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6f02c360-27f4-4a25-8155-436e260be559-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wx4cl\" (UID: \"6f02c360-27f4-4a25-8155-436e260be559\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wx4cl"
Oct 30 22:23:47.149012 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:47.148995    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6f02c360-27f4-4a25-8155-436e260be559-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wx4cl\" (UID: \"6f02c360-27f4-4a25-8155-436e260be559\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wx4cl"
Oct 30 22:23:47.163747 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:47.163721    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f02c360-27f4-4a25-8155-436e260be559-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wx4cl\" (UID: \"6f02c360-27f4-4a25-8155-436e260be559\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wx4cl"
Oct 30 22:23:47.166440 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:47.165090    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gk9dp\" (UniqueName: \"kubernetes.io/projected/6f02c360-27f4-4a25-8155-436e260be559-kube-api-access-gk9dp\") pod \"istiod-openshift-gateway-7ddd9cfdc8-wx4cl\" (UID: \"6f02c360-27f4-4a25-8155-436e260be559\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wx4cl"
Oct 30 22:23:47.281162 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:47.281121    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ca259734-7f54c968d7-v4pjb" event={"ID":"f4b04c98-97ff-4c2c-bdf9-45d5e7e2006e","Type":"ContainerStarted","Data":"8e25a6945b706beb92dc37c801951c8dee345bac31b523ef5efcf7682a0b3bbe"}
Oct 30 22:23:47.281362 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:47.281171    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ca259734-7f54c968d7-v4pjb" event={"ID":"f4b04c98-97ff-4c2c-bdf9-45d5e7e2006e","Type":"ContainerStarted","Data":"cd167bd74ac79207de2be62e6a164d6aca9b50988c59c078673d5fc36f4fa9ca"}
Oct 30 22:23:47.281452 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:47.281358    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ca259734-7f54c968d7-v4pjb"
Oct 30 22:23:47.283554 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:47.283532    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wx4cl"
Oct 30 22:23:47.318901 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:47.318810    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ca259734-7f54c968d7-v4pjb" podStartSLOduration=2.967938743 podStartE2EDuration="3.318793416s" totalImagesPullingTime="350.854673ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:23:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:23:47.317952811 +0000 UTC m=+1011.243879031" watchObservedRunningTime="2026-10-30 22:23:47.318793416 +0000 UTC m=+1011.244719625"
Oct 30 22:23:47.594118 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:47.592214    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7969e73-ec10-41a1-a26f-6477c17e55fb-maas-api-tls\") pod \"maas-api-e2e-ait-1ba9ae08-999dc7c5d-wpffx\" (UID: \"d7969e73-ec10-41a1-a26f-6477c17e55fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1ba9ae08-999dc7c5d-wpffx"
Oct 30 22:23:47.594118 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:23:47.592398    4477 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-1ba9ae08-serving-cert: secret "maas-api-e2e-ait-1ba9ae08-serving-cert" not found
Oct 30 22:23:47.594118 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:23:47.592460    4477 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7969e73-ec10-41a1-a26f-6477c17e55fb-maas-api-tls podName:d7969e73-ec10-41a1-a26f-6477c17e55fb nodeName:}" failed. No retries permitted until 2026-10-30 22:23:48.592442131 +0000 UTC m=+1012.518368339 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d7969e73-ec10-41a1-a26f-6477c17e55fb-maas-api-tls") pod "maas-api-e2e-ait-1ba9ae08-999dc7c5d-wpffx" (UID: "d7969e73-ec10-41a1-a26f-6477c17e55fb") : secret "maas-api-e2e-ait-1ba9ae08-serving-cert" not found
Oct 30 22:23:47.663781 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:47.658372    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wx4cl"]
Oct 30 22:23:48.305287 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:48.305242    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wx4cl" event={"ID":"6f02c360-27f4-4a25-8155-436e260be559","Type":"ContainerStarted","Data":"3d57ff0889f1bf6b0d9bb7a76e6eb9bb561a9be67520d70a4b5c0fdc9b616be3"}
Oct 30 22:23:48.340226 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:48.338373    4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-6wn2g"
Oct 30 22:23:48.340226 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:48.340055    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-6wn2g"
Oct 30 22:23:48.347820 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:48.346375    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-6wn2g"
Oct 30 22:23:48.347820 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:48.346975    4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-n6fb8"
Oct 30 22:23:48.348441 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:48.348122    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-n6fb8"
Oct 30 22:23:48.356557 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:48.356536    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-n6fb8"
Oct 30 22:23:48.640909 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:48.640871    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7969e73-ec10-41a1-a26f-6477c17e55fb-maas-api-tls\") pod \"maas-api-e2e-ait-1ba9ae08-999dc7c5d-wpffx\" (UID: \"d7969e73-ec10-41a1-a26f-6477c17e55fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1ba9ae08-999dc7c5d-wpffx"
Oct 30 22:23:48.643288 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:48.643261    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7969e73-ec10-41a1-a26f-6477c17e55fb-maas-api-tls\") pod \"maas-api-e2e-ait-1ba9ae08-999dc7c5d-wpffx\" (UID: \"d7969e73-ec10-41a1-a26f-6477c17e55fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1ba9ae08-999dc7c5d-wpffx"
Oct 30 22:23:48.878088 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:48.878033    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1ba9ae08-999dc7c5d-wpffx"
Oct 30 22:23:49.199704 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:49.199670    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1ba9ae08-999dc7c5d-wpffx"]
Oct 30 22:23:49.202352 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:23:49.202323    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7969e73_ec10_41a1_a26f_6477c17e55fb.slice/crio-77575c1e06de46a7a93e96bde13c7666fbdae575f0041e8a3eb1687b722d2854 WatchSource:0}: Error finding container 77575c1e06de46a7a93e96bde13c7666fbdae575f0041e8a3eb1687b722d2854: Status 404 returned error can't find the container with id 77575c1e06de46a7a93e96bde13c7666fbdae575f0041e8a3eb1687b722d2854
Oct 30 22:23:49.315491 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:49.315447    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1ba9ae08-999dc7c5d-wpffx" event={"ID":"d7969e73-ec10-41a1-a26f-6477c17e55fb","Type":"ContainerStarted","Data":"77575c1e06de46a7a93e96bde13c7666fbdae575f0041e8a3eb1687b722d2854"}
Oct 30 22:23:50.325177 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:50.325134    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1ba9ae08-999dc7c5d-wpffx" event={"ID":"d7969e73-ec10-41a1-a26f-6477c17e55fb","Type":"ContainerStarted","Data":"0a6dd7d371da9c01961aa89a478ad2d47a8d77e8d806317021e2d373273e7631"}
Oct 30 22:23:50.325728 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:50.325321    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1ba9ae08-999dc7c5d-wpffx"
Oct 30 22:23:50.367342 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:50.367287    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1ba9ae08-999dc7c5d-wpffx" podStartSLOduration=4.023265195 podStartE2EDuration="4.367269175s" totalImagesPullingTime="344.00398ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:23:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:23:50.362390951 +0000 UTC m=+1014.288317158" watchObservedRunningTime="2026-10-30 22:23:50.367269175 +0000 UTC m=+1014.293195381"
Oct 30 22:23:50.504610 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:50.504537    4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:23:50.504762 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:50.504616    4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:23:51.335204 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:51.335164    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wx4cl" event={"ID":"6f02c360-27f4-4a25-8155-436e260be559","Type":"ContainerStarted","Data":"00ab2f2b61e429ca6106b222f82cbe4953e90e094ea2fd809234bf5b12308b54"}
Oct 30 22:23:51.335706 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:51.335420    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wx4cl"
Oct 30 22:23:51.337511 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:51.337483    4477 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-wx4cl container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:23:51.337649 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:51.337530    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wx4cl" podUID="6f02c360-27f4-4a25-8155-436e260be559" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:23:51.386834 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:51.386783    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wx4cl" podStartSLOduration=2.532616211 podStartE2EDuration="5.38676866s" totalImagesPullingTime="2.854152449s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:23:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:23:51.383905161 +0000 UTC m=+1015.309831392" watchObservedRunningTime="2026-10-30 22:23:51.38676866 +0000 UTC m=+1015.312694866"
Oct 30 22:23:52.342429 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:52.342400    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-wx4cl"
Oct 30 22:23:53.310806 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:53.310774    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ca259734-7f54c968d7-v4pjb"
Oct 30 22:23:56.270343 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:23:56.270300    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 22:23:56.340289 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:23:56.340258    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1ba9ae08-999dc7c5d-wpffx"
Oct 30 22:24:06.994937 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:06.994905    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6bz92"]
Oct 30 22:24:09.334238 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:09.326674    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/distinct-2-w6-012ba0-kserve-78dbcd8c46-hqdfm" podUID="8326b014-989b-4722-befb-6fa8ea60688b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Oct 30 22:24:09.449007 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:09.448958    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/simulator-w1-2b1e88-kserve-6c579b99f8-q5wpq" podUID="722f10d8-b7a2-4bf7-a274-9669529ef427" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:24:09.517172 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:09.516135    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-2-w4-0ad7c5-kserve-77c5467fcb-skspk" podUID="430bd879-2a89-4070-bb1b-9b9958a261b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:24:09.948470 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:09.948427    4477 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-2-w4-0ad7c5-kserve-77c5467fcb-skspk" podUID="430bd879-2a89-4070-bb1b-9b9958a261b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:24:10.219306 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:10.218124    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-421583/distinct-w2-421583-kserve-69cf5f4d6b-qbgnt" podUID="510ad10d-0554-47c8-9b55-deb359795071" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:24:10.525469 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:10.515341    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/premium-w5-54114e-kserve-57f6747bcc-7wjrw" podUID="61990d22-ab8f-45bc-a4d6-4c8d528742f1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:24:10.525469 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:10.517984    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/distinct-2-w0-09d0d5-kserve-7dcf89b98b-7cjph" podUID="8cbd04b4-bb3e-4bbb-a41a-46131c9de051" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:24:10.562869 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:10.528260    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-2-w4-0ad7c5-kserve-77c5467fcb-skspk" podUID="430bd879-2a89-4070-bb1b-9b9958a261b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:24:10.562869 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:10.528535    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-2-w4-0ad7c5-kserve-77c5467fcb-skspk"
Oct 30 22:24:11.325557 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:11.320397    4477 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-0ad7c5/premium-w4-0ad7c5-kserve-7bbf78cd4-pm6wc" podUID="5c10371e-56e7-42ec-9ef8-0802e15cb2ff" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:24:11.351135 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:11.350140    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-012ba0/distinct-2-w6-012ba0-kserve-78dbcd8c46-hqdfm" podUID="8326b014-989b-4722-befb-6fa8ea60688b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:24:11.541106 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:11.539528    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-54114e/premium-w5-54114e-kserve-57f6747bcc-7wjrw" podUID="61990d22-ab8f-45bc-a4d6-4c8d528742f1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:24:11.541106 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:11.539891    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-54114e/premium-w5-54114e-kserve-57f6747bcc-7wjrw"
Oct 30 22:24:11.552627 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:11.549683    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-2-w4-0ad7c5-kserve-77c5467fcb-skspk" podUID="430bd879-2a89-4070-bb1b-9b9958a261b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:24:11.552627 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:11.551376    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/simulator-w1-2b1e88-kserve-6c579b99f8-q5wpq" podUID="722f10d8-b7a2-4bf7-a274-9669529ef427" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:24:11.552627 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:11.551630    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-09d0d5/distinct-2-w0-09d0d5-kserve-7dcf89b98b-7cjph" podUID="8cbd04b4-bb3e-4bbb-a41a-46131c9de051" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:24:11.552627 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:11.551906    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-09d0d5/distinct-2-w0-09d0d5-kserve-7dcf89b98b-7cjph"
Oct 30 22:24:11.665672 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:11.665643    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-54114e/premium-w5-54114e-kserve-57f6747bcc-7wjrw"
Oct 30 22:24:11.760580 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:11.760304    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-09d0d5/distinct-2-w0-09d0d5-kserve-7dcf89b98b-7cjph"
Oct 30 22:24:11.791108 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:11.777262    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-2-w4-0ad7c5-kserve-77c5467fcb-skspk"
Oct 30 22:24:20.037863 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:20.037830    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be124eb7-dttc7"]
Oct 30 22:24:20.047869 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:20.047842    4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be124eb7-dttc7"]
Oct 30 22:24:20.198304 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:20.198272    4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e886f4d-e60f-4668-b3b6-62baa7f0b185" path="/var/lib/kubelet/pods/0e886f4d-e60f-4668-b3b6-62baa7f0b185/volumes"
Oct 30 22:24:26.266869 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:24:26.266831    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 22:24:30.200055 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:30.200005    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ca259734-6d97f789c5-dfxt8"]
Oct 30 22:24:30.237464 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:30.237420    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ca259734-86c66dfcf9-5dzzk"]
Oct 30 22:24:33.035780 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:33.035742    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc038a41-bsw8z"]
Oct 30 22:24:33.045156 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:33.045131    4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc038a41-bsw8z"]
Oct 30 22:24:33.287617 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:33.287537    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ca259734-7f54c968d7-v4pjb"]
Oct 30 22:24:33.287784 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:33.287763    4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ca259734-7f54c968d7-v4pjb" podUID="f4b04c98-97ff-4c2c-bdf9-45d5e7e2006e" containerName="maas-api" containerID="cri-o://8e25a6945b706beb92dc37c801951c8dee345bac31b523ef5efcf7682a0b3bbe" gracePeriod=30
Oct 30 22:24:33.307496 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:33.307460    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ca259734-7f54c968d7-v4pjb" podUID="f4b04c98-97ff-4c2c-bdf9-45d5e7e2006e" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.83:8443/health\": dial tcp 10.134.0.83:8443: connect: connection refused"
Oct 30 22:24:33.719994 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:33.719959    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ca259734-7f54c968d7-v4pjb"
Oct 30 22:24:33.833621 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:33.832184    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4b04c98-97ff-4c2c-bdf9-45d5e7e2006e-kube-api-access-ctfzc\" (UniqueName: \"kubernetes.io/projected/f4b04c98-97ff-4c2c-bdf9-45d5e7e2006e-kube-api-access-ctfzc\") pod \"f4b04c98-97ff-4c2c-bdf9-45d5e7e2006e\" (UID: \"f4b04c98-97ff-4c2c-bdf9-45d5e7e2006e\") "
Oct 30 22:24:33.833621 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:33.832287    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4b04c98-97ff-4c2c-bdf9-45d5e7e2006e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4b04c98-97ff-4c2c-bdf9-45d5e7e2006e-metrics-certs\") pod \"f4b04c98-97ff-4c2c-bdf9-45d5e7e2006e\" (UID: \"f4b04c98-97ff-4c2c-bdf9-45d5e7e2006e\") "
Oct 30 22:24:33.833621 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:33.832381    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4b04c98-97ff-4c2c-bdf9-45d5e7e2006e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4b04c98-97ff-4c2c-bdf9-45d5e7e2006e-maas-api-tls\") pod \"f4b04c98-97ff-4c2c-bdf9-45d5e7e2006e\" (UID: \"f4b04c98-97ff-4c2c-bdf9-45d5e7e2006e\") "
Oct 30 22:24:33.833621 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:33.832421    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4b04c98-97ff-4c2c-bdf9-45d5e7e2006e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4b04c98-97ff-4c2c-bdf9-45d5e7e2006e-openshift-service-ca\") pod \"f4b04c98-97ff-4c2c-bdf9-45d5e7e2006e\" (UID: \"f4b04c98-97ff-4c2c-bdf9-45d5e7e2006e\") "
Oct 30 22:24:33.861835 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:33.834377    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4b04c98-97ff-4c2c-bdf9-45d5e7e2006e-openshift-service-ca" pod "f4b04c98-97ff-4c2c-bdf9-45d5e7e2006e" (UID: "f4b04c98-97ff-4c2c-bdf9-45d5e7e2006e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:24:33.861835 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:33.834617    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1ba9ae08-999dc7c5d-wpffx"]
Oct 30 22:24:33.861835 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:33.834840    4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1ba9ae08-999dc7c5d-wpffx" podUID="d7969e73-ec10-41a1-a26f-6477c17e55fb" containerName="maas-api" containerID="cri-o://0a6dd7d371da9c01961aa89a478ad2d47a8d77e8d806317021e2d373273e7631" gracePeriod=30
Oct 30 22:24:33.861835 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:33.859671    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4b04c98-97ff-4c2c-bdf9-45d5e7e2006e-kube-api-access-ctfzc" pod "f4b04c98-97ff-4c2c-bdf9-45d5e7e2006e" (UID: "f4b04c98-97ff-4c2c-bdf9-45d5e7e2006e"). InnerVolumeSpecName "kube-api-access-ctfzc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:24:33.861835 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:33.859786    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4b04c98-97ff-4c2c-bdf9-45d5e7e2006e-metrics-certs" pod "f4b04c98-97ff-4c2c-bdf9-45d5e7e2006e" (UID: "f4b04c98-97ff-4c2c-bdf9-45d5e7e2006e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:24:33.861835 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:33.861543    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4b04c98-97ff-4c2c-bdf9-45d5e7e2006e-maas-api-tls" pod "f4b04c98-97ff-4c2c-bdf9-45d5e7e2006e" (UID: "f4b04c98-97ff-4c2c-bdf9-45d5e7e2006e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:24:33.940258 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:33.935276    4477 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4b04c98-97ff-4c2c-bdf9-45d5e7e2006e-openshift-service-ca\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:24:33.940258 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:33.935310    4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctfzc\" (UniqueName: \"kubernetes.io/projected/f4b04c98-97ff-4c2c-bdf9-45d5e7e2006e-kube-api-access-ctfzc\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:24:33.940258 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:33.935324    4477 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4b04c98-97ff-4c2c-bdf9-45d5e7e2006e-metrics-certs\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:24:33.940258 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:33.935340    4477 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4b04c98-97ff-4c2c-bdf9-45d5e7e2006e-maas-api-tls\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:24:34.130619 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:34.130415    4477 generic.go:356] "Generic (PLEG): container finished" podID="d7969e73-ec10-41a1-a26f-6477c17e55fb" containerID="0a6dd7d371da9c01961aa89a478ad2d47a8d77e8d806317021e2d373273e7631" exitCode=0
Oct 30 22:24:34.130619 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:34.130542    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1ba9ae08-999dc7c5d-wpffx" event={"ID":"d7969e73-ec10-41a1-a26f-6477c17e55fb","Type":"ContainerDied","Data":"0a6dd7d371da9c01961aa89a478ad2d47a8d77e8d806317021e2d373273e7631"}
Oct 30 22:24:34.138795 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:34.134362    4477 generic.go:356] "Generic (PLEG): container finished" podID="f4b04c98-97ff-4c2c-bdf9-45d5e7e2006e" containerID="8e25a6945b706beb92dc37c801951c8dee345bac31b523ef5efcf7682a0b3bbe" exitCode=0
Oct 30 22:24:34.138795 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:34.134516    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ca259734-7f54c968d7-v4pjb" event={"ID":"f4b04c98-97ff-4c2c-bdf9-45d5e7e2006e","Type":"ContainerDied","Data":"8e25a6945b706beb92dc37c801951c8dee345bac31b523ef5efcf7682a0b3bbe"}
Oct 30 22:24:34.138795 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:34.134563    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ca259734-7f54c968d7-v4pjb" event={"ID":"f4b04c98-97ff-4c2c-bdf9-45d5e7e2006e","Type":"ContainerDied","Data":"cd167bd74ac79207de2be62e6a164d6aca9b50988c59c078673d5fc36f4fa9ca"}
Oct 30 22:24:34.138795 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:34.134591    4477 scope.go:122] "RemoveContainer" containerID="8e25a6945b706beb92dc37c801951c8dee345bac31b523ef5efcf7682a0b3bbe"
Oct 30 22:24:34.138795 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:34.134758    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ca259734-7f54c968d7-v4pjb"
Oct 30 22:24:34.161991 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:34.161576    4477 scope.go:122] "RemoveContainer" containerID="8e25a6945b706beb92dc37c801951c8dee345bac31b523ef5efcf7682a0b3bbe"
Oct 30 22:24:34.161991 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:24:34.161894    4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e25a6945b706beb92dc37c801951c8dee345bac31b523ef5efcf7682a0b3bbe\": container with ID starting with 8e25a6945b706beb92dc37c801951c8dee345bac31b523ef5efcf7682a0b3bbe not found: ID does not exist" containerID="8e25a6945b706beb92dc37c801951c8dee345bac31b523ef5efcf7682a0b3bbe"
Oct 30 22:24:34.161991 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:34.161931    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e25a6945b706beb92dc37c801951c8dee345bac31b523ef5efcf7682a0b3bbe"} err="failed to get container status \"8e25a6945b706beb92dc37c801951c8dee345bac31b523ef5efcf7682a0b3bbe\": rpc error: code = NotFound desc = could not find container \"8e25a6945b706beb92dc37c801951c8dee345bac31b523ef5efcf7682a0b3bbe\": container with ID starting with 8e25a6945b706beb92dc37c801951c8dee345bac31b523ef5efcf7682a0b3bbe not found: ID does not exist"
Oct 30 22:24:34.190399 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:34.190377    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1ba9ae08-999dc7c5d-wpffx"
Oct 30 22:24:34.203781 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:34.203755    4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa2d3862-6aa7-4f8b-be74-f2f48c2c520a" path="/var/lib/kubelet/pods/aa2d3862-6aa7-4f8b-be74-f2f48c2c520a/volumes"
Oct 30 22:24:34.204108 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:34.204086    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ca259734-7f54c968d7-v4pjb"]
Oct 30 22:24:34.206916 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:34.206891    4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ca259734-7f54c968d7-v4pjb"]
Oct 30 22:24:34.385108 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:34.384993    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7969e73-ec10-41a1-a26f-6477c17e55fb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7969e73-ec10-41a1-a26f-6477c17e55fb-maas-api-tls\") pod \"d7969e73-ec10-41a1-a26f-6477c17e55fb\" (UID: \"d7969e73-ec10-41a1-a26f-6477c17e55fb\") "
Oct 30 22:24:34.385108 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:34.385078    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7969e73-ec10-41a1-a26f-6477c17e55fb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7969e73-ec10-41a1-a26f-6477c17e55fb-metrics-certs\") pod \"d7969e73-ec10-41a1-a26f-6477c17e55fb\" (UID: \"d7969e73-ec10-41a1-a26f-6477c17e55fb\") "
Oct 30 22:24:34.385351 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:34.385324    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7969e73-ec10-41a1-a26f-6477c17e55fb-kube-api-access-vkbqp\" (UniqueName: \"kubernetes.io/projected/d7969e73-ec10-41a1-a26f-6477c17e55fb-kube-api-access-vkbqp\") pod \"d7969e73-ec10-41a1-a26f-6477c17e55fb\" (UID: \"d7969e73-ec10-41a1-a26f-6477c17e55fb\") "
Oct 30 22:24:34.385415 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:34.385368    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7969e73-ec10-41a1-a26f-6477c17e55fb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7969e73-ec10-41a1-a26f-6477c17e55fb-openshift-service-ca\") pod \"d7969e73-ec10-41a1-a26f-6477c17e55fb\" (UID: \"d7969e73-ec10-41a1-a26f-6477c17e55fb\") "
Oct 30 22:24:34.386102 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:34.386027    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7969e73-ec10-41a1-a26f-6477c17e55fb-openshift-service-ca" pod "d7969e73-ec10-41a1-a26f-6477c17e55fb" (UID: "d7969e73-ec10-41a1-a26f-6477c17e55fb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:24:34.387506 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:34.387481    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7969e73-ec10-41a1-a26f-6477c17e55fb-metrics-certs" pod "d7969e73-ec10-41a1-a26f-6477c17e55fb" (UID: "d7969e73-ec10-41a1-a26f-6477c17e55fb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:24:34.388967 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:34.388929    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7969e73-ec10-41a1-a26f-6477c17e55fb-kube-api-access-vkbqp" pod "d7969e73-ec10-41a1-a26f-6477c17e55fb" (UID: "d7969e73-ec10-41a1-a26f-6477c17e55fb"). InnerVolumeSpecName "kube-api-access-vkbqp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:24:34.394767 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:34.394724    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7969e73-ec10-41a1-a26f-6477c17e55fb-maas-api-tls" pod "d7969e73-ec10-41a1-a26f-6477c17e55fb" (UID: "d7969e73-ec10-41a1-a26f-6477c17e55fb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:24:34.493096 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:34.490695    4477 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7969e73-ec10-41a1-a26f-6477c17e55fb-openshift-service-ca\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:24:34.493096 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:34.490736    4477 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7969e73-ec10-41a1-a26f-6477c17e55fb-maas-api-tls\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:24:34.493096 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:34.490751    4477 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7969e73-ec10-41a1-a26f-6477c17e55fb-metrics-certs\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:24:34.493096 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:34.490764    4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkbqp\" (UniqueName: \"kubernetes.io/projected/d7969e73-ec10-41a1-a26f-6477c17e55fb-kube-api-access-vkbqp\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:24:35.145759 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:35.145727    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1ba9ae08-999dc7c5d-wpffx"
Oct 30 22:24:35.145759 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:35.145732    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1ba9ae08-999dc7c5d-wpffx" event={"ID":"d7969e73-ec10-41a1-a26f-6477c17e55fb","Type":"ContainerDied","Data":"77575c1e06de46a7a93e96bde13c7666fbdae575f0041e8a3eb1687b722d2854"}
Oct 30 22:24:35.146331 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:35.145838    4477 scope.go:122] "RemoveContainer" containerID="0a6dd7d371da9c01961aa89a478ad2d47a8d77e8d806317021e2d373273e7631"
Oct 30 22:24:35.201416 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:35.201383    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1ba9ae08-999dc7c5d-wpffx"]
Oct 30 22:24:35.201579 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:35.201446    4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-ca259734-6d97f789c5-dfxt8" podUID="aa7d15e2-9f74-48c2-b55f-73c2179b5e4f" containerName="payload-processing" containerID="cri-o://4f7b28839389acbe892db8ecf9b284c71324a063df4bacc0bdd3142b1cd9098e" gracePeriod=25
Oct 30 22:24:35.208674 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:35.208650    4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1ba9ae08-999dc7c5d-wpffx"]
Oct 30 22:24:35.238789 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:35.238747    4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-ca259734-86c66dfcf9-5dzzk" podUID="6ba28fff-2bad-48f3-8d15-f4dc174ce26b" containerName="payload-pre-processing" containerID="cri-o://4bd3aca9dbc3f00fddf33a1296d850d768e89e3590dbb1631789cbab4340281e" gracePeriod=25
Oct 30 22:24:35.648938 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:35.643580    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ca259734-6d97f789c5-dfxt8"
Oct 30 22:24:35.719182 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:35.719156    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa7d15e2-9f74-48c2-b55f-73c2179b5e4f-kube-api-access-nb2lk\" (UniqueName: \"kubernetes.io/projected/aa7d15e2-9f74-48c2-b55f-73c2179b5e4f-kube-api-access-nb2lk\") pod \"aa7d15e2-9f74-48c2-b55f-73c2179b5e4f\" (UID: \"aa7d15e2-9f74-48c2-b55f-73c2179b5e4f\") "
Oct 30 22:24:35.719411 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:35.719392    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa7d15e2-9f74-48c2-b55f-73c2179b5e4f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa7d15e2-9f74-48c2-b55f-73c2179b5e4f-plugins-config-volume\") pod \"aa7d15e2-9f74-48c2-b55f-73c2179b5e4f\" (UID: \"aa7d15e2-9f74-48c2-b55f-73c2179b5e4f\") "
Oct 30 22:24:35.719900 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:35.719863    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa7d15e2-9f74-48c2-b55f-73c2179b5e4f-plugins-config-volume" pod "aa7d15e2-9f74-48c2-b55f-73c2179b5e4f" (UID: "aa7d15e2-9f74-48c2-b55f-73c2179b5e4f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:24:35.720208 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:35.720185    4477 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa7d15e2-9f74-48c2-b55f-73c2179b5e4f-plugins-config-volume\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:24:35.721799 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:35.721766    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa7d15e2-9f74-48c2-b55f-73c2179b5e4f-kube-api-access-nb2lk" pod "aa7d15e2-9f74-48c2-b55f-73c2179b5e4f" (UID: "aa7d15e2-9f74-48c2-b55f-73c2179b5e4f"). InnerVolumeSpecName "kube-api-access-nb2lk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:24:35.753109 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:35.753084    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ca259734-86c66dfcf9-5dzzk"
Oct 30 22:24:35.821339 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:35.821309    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ba28fff-2bad-48f3-8d15-f4dc174ce26b-kube-api-access-tfk26\" (UniqueName: \"kubernetes.io/projected/6ba28fff-2bad-48f3-8d15-f4dc174ce26b-kube-api-access-tfk26\") pod \"6ba28fff-2bad-48f3-8d15-f4dc174ce26b\" (UID: \"6ba28fff-2bad-48f3-8d15-f4dc174ce26b\") "
Oct 30 22:24:35.821494 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:35.821386    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ba28fff-2bad-48f3-8d15-f4dc174ce26b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ba28fff-2bad-48f3-8d15-f4dc174ce26b-plugins-config-volume\") pod \"6ba28fff-2bad-48f3-8d15-f4dc174ce26b\" (UID: \"6ba28fff-2bad-48f3-8d15-f4dc174ce26b\") "
Oct 30 22:24:35.821962 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:35.821940    4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nb2lk\" (UniqueName: \"kubernetes.io/projected/aa7d15e2-9f74-48c2-b55f-73c2179b5e4f-kube-api-access-nb2lk\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:24:35.822580 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:35.822552    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ba28fff-2bad-48f3-8d15-f4dc174ce26b-plugins-config-volume" pod "6ba28fff-2bad-48f3-8d15-f4dc174ce26b" (UID: "6ba28fff-2bad-48f3-8d15-f4dc174ce26b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:24:35.824294 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:35.824266    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ba28fff-2bad-48f3-8d15-f4dc174ce26b-kube-api-access-tfk26" pod "6ba28fff-2bad-48f3-8d15-f4dc174ce26b" (UID: "6ba28fff-2bad-48f3-8d15-f4dc174ce26b"). InnerVolumeSpecName "kube-api-access-tfk26". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:24:35.924227 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:35.924187    4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfk26\" (UniqueName: \"kubernetes.io/projected/6ba28fff-2bad-48f3-8d15-f4dc174ce26b-kube-api-access-tfk26\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:24:35.924227 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:35.924218    4477 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ba28fff-2bad-48f3-8d15-f4dc174ce26b-plugins-config-volume\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:24:36.161371 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:36.161336    4477 generic.go:356] "Generic (PLEG): container finished" podID="aa7d15e2-9f74-48c2-b55f-73c2179b5e4f" containerID="4f7b28839389acbe892db8ecf9b284c71324a063df4bacc0bdd3142b1cd9098e" exitCode=0
Oct 30 22:24:36.162134 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:36.161950    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ca259734-6d97f789c5-dfxt8" event={"ID":"aa7d15e2-9f74-48c2-b55f-73c2179b5e4f","Type":"ContainerDied","Data":"4f7b28839389acbe892db8ecf9b284c71324a063df4bacc0bdd3142b1cd9098e"}
Oct 30 22:24:36.162320 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:36.162301    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ca259734-6d97f789c5-dfxt8" event={"ID":"aa7d15e2-9f74-48c2-b55f-73c2179b5e4f","Type":"ContainerDied","Data":"c63e32364f0f5ead3a0403697e5c30a638633720a889d2f4922b10dcda646103"}
Oct 30 22:24:36.162971 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:36.162420    4477 scope.go:122] "RemoveContainer" containerID="4f7b28839389acbe892db8ecf9b284c71324a063df4bacc0bdd3142b1cd9098e"
Oct 30 22:24:36.164101 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:36.162056    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ca259734-6d97f789c5-dfxt8"
Oct 30 22:24:36.169611 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:36.168484    4477 generic.go:356] "Generic (PLEG): container finished" podID="6ba28fff-2bad-48f3-8d15-f4dc174ce26b" containerID="4bd3aca9dbc3f00fddf33a1296d850d768e89e3590dbb1631789cbab4340281e" exitCode=0
Oct 30 22:24:36.169611 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:36.168653    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ca259734-86c66dfcf9-5dzzk" event={"ID":"6ba28fff-2bad-48f3-8d15-f4dc174ce26b","Type":"ContainerDied","Data":"4bd3aca9dbc3f00fddf33a1296d850d768e89e3590dbb1631789cbab4340281e"}
Oct 30 22:24:36.169611 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:36.168690    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ca259734-86c66dfcf9-5dzzk" event={"ID":"6ba28fff-2bad-48f3-8d15-f4dc174ce26b","Type":"ContainerDied","Data":"82f73d36426fd5378281fae2dab7117ab4181f6af9a0241fd028f7a3c47f9d18"}
Oct 30 22:24:36.169611 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:36.168762    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ca259734-86c66dfcf9-5dzzk"
Oct 30 22:24:36.191555 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:36.191527    4477 scope.go:122] "RemoveContainer" containerID="4f7b28839389acbe892db8ecf9b284c71324a063df4bacc0bdd3142b1cd9098e"
Oct 30 22:24:36.192371 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:24:36.192220    4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f7b28839389acbe892db8ecf9b284c71324a063df4bacc0bdd3142b1cd9098e\": container with ID starting with 4f7b28839389acbe892db8ecf9b284c71324a063df4bacc0bdd3142b1cd9098e not found: ID does not exist" containerID="4f7b28839389acbe892db8ecf9b284c71324a063df4bacc0bdd3142b1cd9098e"
Oct 30 22:24:36.192371 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:36.192261    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f7b28839389acbe892db8ecf9b284c71324a063df4bacc0bdd3142b1cd9098e"} err="failed to get container status \"4f7b28839389acbe892db8ecf9b284c71324a063df4bacc0bdd3142b1cd9098e\": rpc error: code = NotFound desc = could not find container \"4f7b28839389acbe892db8ecf9b284c71324a063df4bacc0bdd3142b1cd9098e\": container with ID starting with 4f7b28839389acbe892db8ecf9b284c71324a063df4bacc0bdd3142b1cd9098e not found: ID does not exist"
Oct 30 22:24:36.192371 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:36.192282    4477 scope.go:122] "RemoveContainer" containerID="4bd3aca9dbc3f00fddf33a1296d850d768e89e3590dbb1631789cbab4340281e"
Oct 30 22:24:36.207505 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:36.207454    4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7969e73-ec10-41a1-a26f-6477c17e55fb" path="/var/lib/kubelet/pods/d7969e73-ec10-41a1-a26f-6477c17e55fb/volumes"
Oct 30 22:24:36.208250 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:36.208182    4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4b04c98-97ff-4c2c-bdf9-45d5e7e2006e" path="/var/lib/kubelet/pods/f4b04c98-97ff-4c2c-bdf9-45d5e7e2006e/volumes"
Oct 30 22:24:36.208399 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:36.208364    4477 scope.go:122] "RemoveContainer" containerID="4bd3aca9dbc3f00fddf33a1296d850d768e89e3590dbb1631789cbab4340281e"
Oct 30 22:24:36.208802 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:24:36.208771    4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bd3aca9dbc3f00fddf33a1296d850d768e89e3590dbb1631789cbab4340281e\": container with ID starting with 4bd3aca9dbc3f00fddf33a1296d850d768e89e3590dbb1631789cbab4340281e not found: ID does not exist" containerID="4bd3aca9dbc3f00fddf33a1296d850d768e89e3590dbb1631789cbab4340281e"
Oct 30 22:24:36.208871 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:36.208811    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bd3aca9dbc3f00fddf33a1296d850d768e89e3590dbb1631789cbab4340281e"} err="failed to get container status \"4bd3aca9dbc3f00fddf33a1296d850d768e89e3590dbb1631789cbab4340281e\": rpc error: code = NotFound desc = could not find container \"4bd3aca9dbc3f00fddf33a1296d850d768e89e3590dbb1631789cbab4340281e\": container with ID starting with 4bd3aca9dbc3f00fddf33a1296d850d768e89e3590dbb1631789cbab4340281e not found: ID does not exist"
Oct 30 22:24:36.246867 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:36.246836    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ca259734-86c66dfcf9-5dzzk"]
Oct 30 22:24:36.251497 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:36.251471    4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ca259734-86c66dfcf9-5dzzk"]
Oct 30 22:24:36.280356 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:36.280315    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ca259734-6d97f789c5-dfxt8"]
Oct 30 22:24:36.284088 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:36.284063    4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ca259734-6d97f789c5-dfxt8"]
Oct 30 22:24:38.207663 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:38.207631    4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ba28fff-2bad-48f3-8d15-f4dc174ce26b" path="/var/lib/kubelet/pods/6ba28fff-2bad-48f3-8d15-f4dc174ce26b/volumes"
Oct 30 22:24:38.208125 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:38.208110    4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa7d15e2-9f74-48c2-b55f-73c2179b5e4f" path="/var/lib/kubelet/pods/aa7d15e2-9f74-48c2-b55f-73c2179b5e4f/volumes"
Oct 30 22:24:46.746550 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:46.746517    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6bz92"]
Oct 30 22:24:48.046030 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:48.046001    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d9b4df1-xrwbz"]
Oct 30 22:24:48.056128 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:48.056103    4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d9b4df1-xrwbz"]
Oct 30 22:24:48.194986 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:48.194958    4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7fd3273-aa8d-466e-a0e6-ec57f8723db6" path="/var/lib/kubelet/pods/a7fd3273-aa8d-466e-a0e6-ec57f8723db6/volumes"
Oct 30 22:24:53.648288 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:53.648257    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-476e593d-576dd78ccf-sktt5"]
Oct 30 22:24:53.649094 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:53.649072    4477 cpu_manager.go:414] "containerMap: removing container" podUID="aa7d15e2-9f74-48c2-b55f-73c2179b5e4f" containerName="payload-processing"
Oct 30 22:24:53.649094 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:53.649093    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa7d15e2-9f74-48c2-b55f-73c2179b5e4f" containerName="payload-processing"
Oct 30 22:24:53.649262 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:53.649123    4477 cpu_manager.go:414] "containerMap: removing container" podUID="6ba28fff-2bad-48f3-8d15-f4dc174ce26b" containerName="payload-pre-processing"
Oct 30 22:24:53.649262 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:53.649131    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ba28fff-2bad-48f3-8d15-f4dc174ce26b" containerName="payload-pre-processing"
Oct 30 22:24:53.649262 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:53.649167    4477 cpu_manager.go:414] "containerMap: removing container" podUID="f4b04c98-97ff-4c2c-bdf9-45d5e7e2006e" containerName="maas-api"
Oct 30 22:24:53.649262 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:53.649174    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4b04c98-97ff-4c2c-bdf9-45d5e7e2006e" containerName="maas-api"
Oct 30 22:24:53.649262 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:53.649198    4477 cpu_manager.go:414] "containerMap: removing container" podUID="d7969e73-ec10-41a1-a26f-6477c17e55fb" containerName="maas-api"
Oct 30 22:24:53.649262 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:53.649205    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7969e73-ec10-41a1-a26f-6477c17e55fb" containerName="maas-api"
Oct 30 22:24:53.649553 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:53.649360    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7969e73-ec10-41a1-a26f-6477c17e55fb" containerName="maas-api"
Oct 30 22:24:53.649553 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:53.649379    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa7d15e2-9f74-48c2-b55f-73c2179b5e4f" containerName="payload-processing"
Oct 30 22:24:53.649553 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:53.649393    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ba28fff-2bad-48f3-8d15-f4dc174ce26b" containerName="payload-pre-processing"
Oct 30 22:24:53.649553 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:53.649407    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4b04c98-97ff-4c2c-bdf9-45d5e7e2006e" containerName="maas-api"
Oct 30 22:24:53.653198 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:53.653179    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-476e593d-576dd78ccf-sktt5"
Oct 30 22:24:53.680502 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:53.680477    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-476e593d-serving-cert\""
Oct 30 22:24:53.684365 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:53.684341    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-476e593d-576dd78ccf-sktt5"]
Oct 30 22:24:53.741727 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:53.741702    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a71419a-bd60-46e5-8083-1dff19349b30-metrics-certs\") pod \"maas-api-e2e-migrate-476e593d-576dd78ccf-sktt5\" (UID: \"3a71419a-bd60-46e5-8083-1dff19349b30\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-476e593d-576dd78ccf-sktt5"
Oct 30 22:24:53.741904 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:53.741885    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a71419a-bd60-46e5-8083-1dff19349b30-maas-api-tls\") pod \"maas-api-e2e-migrate-476e593d-576dd78ccf-sktt5\" (UID: \"3a71419a-bd60-46e5-8083-1dff19349b30\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-476e593d-576dd78ccf-sktt5"
Oct 30 22:24:53.741966 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:53.741931    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ql4tf\" (UniqueName: \"kubernetes.io/projected/3a71419a-bd60-46e5-8083-1dff19349b30-kube-api-access-ql4tf\") pod \"maas-api-e2e-migrate-476e593d-576dd78ccf-sktt5\" (UID: \"3a71419a-bd60-46e5-8083-1dff19349b30\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-476e593d-576dd78ccf-sktt5"
Oct 30 22:24:53.742033 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:53.742020    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a71419a-bd60-46e5-8083-1dff19349b30-openshift-service-ca\") pod \"maas-api-e2e-migrate-476e593d-576dd78ccf-sktt5\" (UID: \"3a71419a-bd60-46e5-8083-1dff19349b30\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-476e593d-576dd78ccf-sktt5"
Oct 30 22:24:53.853065 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:53.844363    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a71419a-bd60-46e5-8083-1dff19349b30-openshift-service-ca\") pod \"maas-api-e2e-migrate-476e593d-576dd78ccf-sktt5\" (UID: \"3a71419a-bd60-46e5-8083-1dff19349b30\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-476e593d-576dd78ccf-sktt5"
Oct 30 22:24:53.853065 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:53.845255    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a71419a-bd60-46e5-8083-1dff19349b30-metrics-certs\") pod \"maas-api-e2e-migrate-476e593d-576dd78ccf-sktt5\" (UID: \"3a71419a-bd60-46e5-8083-1dff19349b30\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-476e593d-576dd78ccf-sktt5"
Oct 30 22:24:53.853065 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:53.845485    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a71419a-bd60-46e5-8083-1dff19349b30-maas-api-tls\") pod \"maas-api-e2e-migrate-476e593d-576dd78ccf-sktt5\" (UID: \"3a71419a-bd60-46e5-8083-1dff19349b30\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-476e593d-576dd78ccf-sktt5"
Oct 30 22:24:53.853065 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:53.845545    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ql4tf\" (UniqueName: \"kubernetes.io/projected/3a71419a-bd60-46e5-8083-1dff19349b30-kube-api-access-ql4tf\") pod \"maas-api-e2e-migrate-476e593d-576dd78ccf-sktt5\" (UID: \"3a71419a-bd60-46e5-8083-1dff19349b30\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-476e593d-576dd78ccf-sktt5"
Oct 30 22:24:53.853065 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:53.845129    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a71419a-bd60-46e5-8083-1dff19349b30-openshift-service-ca\") pod \"maas-api-e2e-migrate-476e593d-576dd78ccf-sktt5\" (UID: \"3a71419a-bd60-46e5-8083-1dff19349b30\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-476e593d-576dd78ccf-sktt5"
Oct 30 22:24:53.853065 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:53.849224    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a71419a-bd60-46e5-8083-1dff19349b30-maas-api-tls\") pod \"maas-api-e2e-migrate-476e593d-576dd78ccf-sktt5\" (UID: \"3a71419a-bd60-46e5-8083-1dff19349b30\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-476e593d-576dd78ccf-sktt5"
Oct 30 22:24:53.853065 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:53.850895    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a71419a-bd60-46e5-8083-1dff19349b30-metrics-certs\") pod \"maas-api-e2e-migrate-476e593d-576dd78ccf-sktt5\" (UID: \"3a71419a-bd60-46e5-8083-1dff19349b30\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-476e593d-576dd78ccf-sktt5"
Oct 30 22:24:53.869589 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:53.865245    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ql4tf\" (UniqueName: \"kubernetes.io/projected/3a71419a-bd60-46e5-8083-1dff19349b30-kube-api-access-ql4tf\") pod \"maas-api-e2e-migrate-476e593d-576dd78ccf-sktt5\" (UID: \"3a71419a-bd60-46e5-8083-1dff19349b30\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-476e593d-576dd78ccf-sktt5"
Oct 30 22:24:53.966322 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:53.966239    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-476e593d-576dd78ccf-sktt5"
Oct 30 22:24:54.281604 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:54.281564    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-476e593d-576dd78ccf-sktt5"]
Oct 30 22:24:54.284888 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:24:54.284853    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a71419a_bd60_46e5_8083_1dff19349b30.slice/crio-73d81304076e8a79103ba4c9f359d52674fd33f9414cbc4139ca56d737bdc65f WatchSource:0}: Error finding container 73d81304076e8a79103ba4c9f359d52674fd33f9414cbc4139ca56d737bdc65f: Status 404 returned error can't find the container with id 73d81304076e8a79103ba4c9f359d52674fd33f9414cbc4139ca56d737bdc65f
Oct 30 22:24:54.348103 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:54.348073    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-476e593d-576dd78ccf-sktt5" event={"ID":"3a71419a-bd60-46e5-8083-1dff19349b30","Type":"ContainerStarted","Data":"73d81304076e8a79103ba4c9f359d52674fd33f9414cbc4139ca56d737bdc65f"}
Oct 30 22:24:55.356205 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:55.356165    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-476e593d-576dd78ccf-sktt5" event={"ID":"3a71419a-bd60-46e5-8083-1dff19349b30","Type":"ContainerStarted","Data":"674f15d9573ede73e4546e026c7d84f04ebda48fdc3e103b1b42ece6eff76e9a"}
Oct 30 22:24:55.356625 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:55.356421    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-476e593d-576dd78ccf-sktt5"
Oct 30 22:24:55.396568 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:24:55.396524    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-476e593d-576dd78ccf-sktt5" podStartSLOduration=2.080410529 podStartE2EDuration="2.39650545s" totalImagesPullingTime="316.094921ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:24:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:24:55.392427797 +0000 UTC m=+1079.318354020" watchObservedRunningTime="2026-10-30 22:24:55.39650545 +0000 UTC m=+1079.322431656"
Oct 30 22:24:56.266979 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:24:56.266945    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 22:25:01.367616 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:25:01.367587    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-476e593d-576dd78ccf-sktt5"
Oct 30 22:25:23.446205 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:25:23.446170    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6bz92"]
Oct 30 22:25:26.276546 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:25:26.276495    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 22:25:32.132783 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:25:32.132752    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-476e593d-576dd78ccf-sktt5"]
Oct 30 22:25:32.133509 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:25:32.132950    4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-476e593d-576dd78ccf-sktt5" podUID="3a71419a-bd60-46e5-8083-1dff19349b30" containerName="maas-api" containerID="cri-o://674f15d9573ede73e4546e026c7d84f04ebda48fdc3e103b1b42ece6eff76e9a" gracePeriod=30
Oct 30 22:25:32.439261 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:25:32.439231    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-476e593d-576dd78ccf-sktt5"
Oct 30 22:25:32.568655 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:25:32.567758    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a71419a-bd60-46e5-8083-1dff19349b30-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a71419a-bd60-46e5-8083-1dff19349b30-metrics-certs\") pod \"3a71419a-bd60-46e5-8083-1dff19349b30\" (UID: \"3a71419a-bd60-46e5-8083-1dff19349b30\") "
Oct 30 22:25:32.568655 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:25:32.567898    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a71419a-bd60-46e5-8083-1dff19349b30-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a71419a-bd60-46e5-8083-1dff19349b30-maas-api-tls\") pod \"3a71419a-bd60-46e5-8083-1dff19349b30\" (UID: \"3a71419a-bd60-46e5-8083-1dff19349b30\") "
Oct 30 22:25:32.568655 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:25:32.567948    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a71419a-bd60-46e5-8083-1dff19349b30-kube-api-access-ql4tf\" (UniqueName: \"kubernetes.io/projected/3a71419a-bd60-46e5-8083-1dff19349b30-kube-api-access-ql4tf\") pod \"3a71419a-bd60-46e5-8083-1dff19349b30\" (UID: \"3a71419a-bd60-46e5-8083-1dff19349b30\") "
Oct 30 22:25:32.568655 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:25:32.568178    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a71419a-bd60-46e5-8083-1dff19349b30-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a71419a-bd60-46e5-8083-1dff19349b30-openshift-service-ca\") pod \"3a71419a-bd60-46e5-8083-1dff19349b30\" (UID: \"3a71419a-bd60-46e5-8083-1dff19349b30\") "
Oct 30 22:25:32.569367 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:25:32.569338    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a71419a-bd60-46e5-8083-1dff19349b30-openshift-service-ca" pod "3a71419a-bd60-46e5-8083-1dff19349b30" (UID: "3a71419a-bd60-46e5-8083-1dff19349b30"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:25:32.574976 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:25:32.574933    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a71419a-bd60-46e5-8083-1dff19349b30-kube-api-access-ql4tf" pod "3a71419a-bd60-46e5-8083-1dff19349b30" (UID: "3a71419a-bd60-46e5-8083-1dff19349b30"). InnerVolumeSpecName "kube-api-access-ql4tf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:25:32.592680 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:25:32.591516    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a71419a-bd60-46e5-8083-1dff19349b30-maas-api-tls" pod "3a71419a-bd60-46e5-8083-1dff19349b30" (UID: "3a71419a-bd60-46e5-8083-1dff19349b30"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:25:32.604282 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:25:32.604190    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a71419a-bd60-46e5-8083-1dff19349b30-metrics-certs" pod "3a71419a-bd60-46e5-8083-1dff19349b30" (UID: "3a71419a-bd60-46e5-8083-1dff19349b30"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:25:32.667461 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:25:32.667357    4477 generic.go:356] "Generic (PLEG): container finished" podID="3a71419a-bd60-46e5-8083-1dff19349b30" containerID="674f15d9573ede73e4546e026c7d84f04ebda48fdc3e103b1b42ece6eff76e9a" exitCode=0
Oct 30 22:25:32.667461 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:25:32.667447    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-476e593d-576dd78ccf-sktt5" event={"ID":"3a71419a-bd60-46e5-8083-1dff19349b30","Type":"ContainerDied","Data":"674f15d9573ede73e4546e026c7d84f04ebda48fdc3e103b1b42ece6eff76e9a"}
Oct 30 22:25:32.667674 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:25:32.667497    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-476e593d-576dd78ccf-sktt5" event={"ID":"3a71419a-bd60-46e5-8083-1dff19349b30","Type":"ContainerDied","Data":"73d81304076e8a79103ba4c9f359d52674fd33f9414cbc4139ca56d737bdc65f"}
Oct 30 22:25:32.667674 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:25:32.667527    4477 scope.go:122] "RemoveContainer" containerID="674f15d9573ede73e4546e026c7d84f04ebda48fdc3e103b1b42ece6eff76e9a"
Oct 30 22:25:32.667674 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:25:32.667583    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-476e593d-576dd78ccf-sktt5"
Oct 30 22:25:32.670192 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:25:32.670167    4477 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3a71419a-bd60-46e5-8083-1dff19349b30-metrics-certs\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:25:32.670192 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:25:32.670197    4477 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3a71419a-bd60-46e5-8083-1dff19349b30-maas-api-tls\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:25:32.670461 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:25:32.670213    4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ql4tf\" (UniqueName: \"kubernetes.io/projected/3a71419a-bd60-46e5-8083-1dff19349b30-kube-api-access-ql4tf\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:25:32.670461 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:25:32.670228    4477 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a71419a-bd60-46e5-8083-1dff19349b30-openshift-service-ca\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:25:32.680932 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:25:32.680880    4477 scope.go:122] "RemoveContainer" containerID="674f15d9573ede73e4546e026c7d84f04ebda48fdc3e103b1b42ece6eff76e9a"
Oct 30 22:25:32.681203 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:25:32.681179    4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"674f15d9573ede73e4546e026c7d84f04ebda48fdc3e103b1b42ece6eff76e9a\": container with ID starting with 674f15d9573ede73e4546e026c7d84f04ebda48fdc3e103b1b42ece6eff76e9a not found: ID does not exist" containerID="674f15d9573ede73e4546e026c7d84f04ebda48fdc3e103b1b42ece6eff76e9a"
Oct 30 22:25:32.681293 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:25:32.681212    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"674f15d9573ede73e4546e026c7d84f04ebda48fdc3e103b1b42ece6eff76e9a"} err="failed to get container status \"674f15d9573ede73e4546e026c7d84f04ebda48fdc3e103b1b42ece6eff76e9a\": rpc error: code = NotFound desc = could not find container \"674f15d9573ede73e4546e026c7d84f04ebda48fdc3e103b1b42ece6eff76e9a\": container with ID starting with 674f15d9573ede73e4546e026c7d84f04ebda48fdc3e103b1b42ece6eff76e9a not found: ID does not exist"
Oct 30 22:25:32.719452 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:25:32.719415    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-476e593d-576dd78ccf-sktt5"]
Oct 30 22:25:32.727860 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:25:32.727812    4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-476e593d-576dd78ccf-sktt5"]
Oct 30 22:25:34.195070 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:25:34.195028    4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a71419a-bd60-46e5-8083-1dff19349b30" path="/var/lib/kubelet/pods/3a71419a-bd60-46e5-8083-1dff19349b30/volumes"
Oct 30 22:25:52.622592 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:25:52.622516    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2949c143-5d858999bf-bwqg9"]
Oct 30 22:25:52.623156 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:25:52.623143    4477 cpu_manager.go:414] "containerMap: removing container" podUID="3a71419a-bd60-46e5-8083-1dff19349b30" containerName="maas-api"
Oct 30 22:25:52.623210 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:25:52.623157    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a71419a-bd60-46e5-8083-1dff19349b30" containerName="maas-api"
Oct 30 22:25:52.623300 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:25:52.623290    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a71419a-bd60-46e5-8083-1dff19349b30" containerName="maas-api"
Oct 30 22:25:52.626789 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:25:52.626773    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2949c143-5d858999bf-bwqg9"
Oct 30 22:25:52.639774 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:25:52.639753    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-2949c143-serving-cert\""
Oct 30 22:25:52.654914 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:25:52.654893    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2949c143-5d858999bf-bwqg9"]
Oct 30 22:25:52.738068 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:25:52.738011    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c16202d5-34f4-4112-9972-b95ebbd47068-maas-api-tls\") pod \"maas-api-e2e-mt-2949c143-5d858999bf-bwqg9\" (UID: \"c16202d5-34f4-4112-9972-b95ebbd47068\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2949c143-5d858999bf-bwqg9"
Oct 30 22:25:52.738214 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:25:52.738083    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c16202d5-34f4-4112-9972-b95ebbd47068-openshift-service-ca\") pod \"maas-api-e2e-mt-2949c143-5d858999bf-bwqg9\" (UID: \"c16202d5-34f4-4112-9972-b95ebbd47068\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2949c143-5d858999bf-bwqg9"
Oct 30 22:25:52.738214 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:25:52.738148    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjwlq\" (UniqueName: \"kubernetes.io/projected/c16202d5-34f4-4112-9972-b95ebbd47068-kube-api-access-hjwlq\") pod \"maas-api-e2e-mt-2949c143-5d858999bf-bwqg9\" (UID: \"c16202d5-34f4-4112-9972-b95ebbd47068\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2949c143-5d858999bf-bwqg9"
Oct 30 22:25:52.738297 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:25:52.738276    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c16202d5-34f4-4112-9972-b95ebbd47068-metrics-certs\") pod \"maas-api-e2e-mt-2949c143-5d858999bf-bwqg9\" (UID: \"c16202d5-34f4-4112-9972-b95ebbd47068\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2949c143-5d858999bf-bwqg9"
Oct 30 22:25:52.839293 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:25:52.839261    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c16202d5-34f4-4112-9972-b95ebbd47068-maas-api-tls\") pod \"maas-api-e2e-mt-2949c143-5d858999bf-bwqg9\" (UID: \"c16202d5-34f4-4112-9972-b95ebbd47068\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2949c143-5d858999bf-bwqg9"
Oct 30 22:25:52.839459 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:25:52.839298    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c16202d5-34f4-4112-9972-b95ebbd47068-openshift-service-ca\") pod \"maas-api-e2e-mt-2949c143-5d858999bf-bwqg9\" (UID: \"c16202d5-34f4-4112-9972-b95ebbd47068\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2949c143-5d858999bf-bwqg9"
Oct 30 22:25:52.839459 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:25:52.839343    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjwlq\" (UniqueName: \"kubernetes.io/projected/c16202d5-34f4-4112-9972-b95ebbd47068-kube-api-access-hjwlq\") pod \"maas-api-e2e-mt-2949c143-5d858999bf-bwqg9\" (UID: \"c16202d5-34f4-4112-9972-b95ebbd47068\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2949c143-5d858999bf-bwqg9"
Oct 30 22:25:52.839567 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:25:52.839467    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c16202d5-34f4-4112-9972-b95ebbd47068-metrics-certs\") pod \"maas-api-e2e-mt-2949c143-5d858999bf-bwqg9\" (UID: \"c16202d5-34f4-4112-9972-b95ebbd47068\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2949c143-5d858999bf-bwqg9"
Oct 30 22:25:52.840148 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:25:52.840123    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c16202d5-34f4-4112-9972-b95ebbd47068-openshift-service-ca\") pod \"maas-api-e2e-mt-2949c143-5d858999bf-bwqg9\" (UID: \"c16202d5-34f4-4112-9972-b95ebbd47068\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2949c143-5d858999bf-bwqg9"
Oct 30 22:25:52.841736 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:25:52.841700    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c16202d5-34f4-4112-9972-b95ebbd47068-maas-api-tls\") pod \"maas-api-e2e-mt-2949c143-5d858999bf-bwqg9\" (UID: \"c16202d5-34f4-4112-9972-b95ebbd47068\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2949c143-5d858999bf-bwqg9"
Oct 30 22:25:52.841843 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:25:52.841758    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c16202d5-34f4-4112-9972-b95ebbd47068-metrics-certs\") pod \"maas-api-e2e-mt-2949c143-5d858999bf-bwqg9\" (UID: \"c16202d5-34f4-4112-9972-b95ebbd47068\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2949c143-5d858999bf-bwqg9"
Oct 30 22:25:52.857965 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:25:52.857944    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjwlq\" (UniqueName: \"kubernetes.io/projected/c16202d5-34f4-4112-9972-b95ebbd47068-kube-api-access-hjwlq\") pod \"maas-api-e2e-mt-2949c143-5d858999bf-bwqg9\" (UID: \"c16202d5-34f4-4112-9972-b95ebbd47068\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2949c143-5d858999bf-bwqg9"
Oct 30 22:25:52.940707 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:25:52.940675    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2949c143-5d858999bf-bwqg9"
Oct 30 22:25:53.252803 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:25:53.252777    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2949c143-5d858999bf-bwqg9"]
Oct 30 22:25:53.255730 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:25:53.255702    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc16202d5_34f4_4112_9972_b95ebbd47068.slice/crio-677baacd575395375a0a758468bf64846e12051c14b259bc71240e6c694c890d WatchSource:0}: Error finding container 677baacd575395375a0a758468bf64846e12051c14b259bc71240e6c694c890d: Status 404 returned error can't find the container with id 677baacd575395375a0a758468bf64846e12051c14b259bc71240e6c694c890d
Oct 30 22:25:53.843346 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:25:53.843314    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2949c143-5d858999bf-bwqg9" event={"ID":"c16202d5-34f4-4112-9972-b95ebbd47068","Type":"ContainerStarted","Data":"82342d31edcfc2dbd1df9dfb3f960ed1720e19f92fbd1a283abf5096e8793511"}
Oct 30 22:25:53.843739 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:25:53.843365    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2949c143-5d858999bf-bwqg9" event={"ID":"c16202d5-34f4-4112-9972-b95ebbd47068","Type":"ContainerStarted","Data":"677baacd575395375a0a758468bf64846e12051c14b259bc71240e6c694c890d"}
Oct 30 22:25:53.843739 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:25:53.843446    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2949c143-5d858999bf-bwqg9"
Oct 30 22:25:53.881018 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:25:53.880970    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2949c143-5d858999bf-bwqg9" podStartSLOduration=1.494463638 podStartE2EDuration="1.880941198s" totalImagesPullingTime="386.47756ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:25:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:25:53.87881084 +0000 UTC m=+1137.804737047" watchObservedRunningTime="2026-10-30 22:25:53.880941198 +0000 UTC m=+1137.806867415"
Oct 30 22:25:56.275514 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:25:56.275252    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 22:25:56.503525 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:25:56.503497    4477 scope.go:122] "RemoveContainer" containerID="ff2c0ab783168598f4f1c21d97e72667f97a385fca7c994c9f8b96dd3027f8db"
Oct 30 22:25:56.523745 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:25:56.523719    4477 scope.go:122] "RemoveContainer" containerID="76ad4847df1bbc11233f262f75dceb46c3a898ebc0e6df2243c4b4045db47639"
Oct 30 22:25:56.535788 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:25:56.535764    4477 scope.go:122] "RemoveContainer" containerID="e161d97fac4599225c9b806cf2745ea9d4746771f5acd0226645341cdfa3b42f"
Oct 30 22:25:59.854966 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:25:59.854941    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2949c143-5d858999bf-bwqg9"
Oct 30 22:26:02.439215 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:26:02.439173    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5983ea-586646597c-z222v"]
Oct 30 22:26:02.444866 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:26:02.444844    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5983ea-586646597c-z222v"
Oct 30 22:26:02.469878 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:26:02.469847    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w0-5983ea-serving-cert\""
Oct 30 22:26:02.474795 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:26:02.474770    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5983ea-586646597c-z222v"]
Oct 30 22:26:02.509085 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:26:02.509007    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/738731d9-e739-4c3e-8936-9c4d01f5fc04-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-5983ea-586646597c-z222v\" (UID: \"738731d9-e739-4c3e-8936-9c4d01f5fc04\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5983ea-586646597c-z222v"
Oct 30 22:26:02.509254 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:26:02.509162    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58lwp\" (UniqueName: \"kubernetes.io/projected/738731d9-e739-4c3e-8936-9c4d01f5fc04-kube-api-access-58lwp\") pod \"maas-api-e2e-shared-a-w0-5983ea-586646597c-z222v\" (UID: \"738731d9-e739-4c3e-8936-9c4d01f5fc04\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5983ea-586646597c-z222v"
Oct 30 22:26:02.509334 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:26:02.509254    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/738731d9-e739-4c3e-8936-9c4d01f5fc04-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-5983ea-586646597c-z222v\" (UID: \"738731d9-e739-4c3e-8936-9c4d01f5fc04\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5983ea-586646597c-z222v"
Oct 30 22:26:02.509448 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:26:02.509421    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/738731d9-e739-4c3e-8936-9c4d01f5fc04-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-5983ea-586646597c-z222v\" (UID: \"738731d9-e739-4c3e-8936-9c4d01f5fc04\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5983ea-586646597c-z222v"
Oct 30 22:26:02.611093 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:26:02.611066    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/738731d9-e739-4c3e-8936-9c4d01f5fc04-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-5983ea-586646597c-z222v\" (UID: \"738731d9-e739-4c3e-8936-9c4d01f5fc04\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5983ea-586646597c-z222v"
Oct 30 22:26:02.611298 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:26:02.611274    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/738731d9-e739-4c3e-8936-9c4d01f5fc04-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-5983ea-586646597c-z222v\" (UID: \"738731d9-e739-4c3e-8936-9c4d01f5fc04\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5983ea-586646597c-z222v"
Oct 30 22:26:02.611375 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:26:02.611348    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58lwp\" (UniqueName: \"kubernetes.io/projected/738731d9-e739-4c3e-8936-9c4d01f5fc04-kube-api-access-58lwp\") pod \"maas-api-e2e-shared-a-w0-5983ea-586646597c-z222v\" (UID: \"738731d9-e739-4c3e-8936-9c4d01f5fc04\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5983ea-586646597c-z222v"
Oct 30 22:26:02.611439 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:26:02.611418    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/738731d9-e739-4c3e-8936-9c4d01f5fc04-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-5983ea-586646597c-z222v\" (UID: \"738731d9-e739-4c3e-8936-9c4d01f5fc04\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5983ea-586646597c-z222v"
Oct 30 22:26:02.611633 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:26:02.611605    4477 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5983ea-serving-cert: secret "maas-api-e2e-shared-a-w0-5983ea-serving-cert" not found
Oct 30 22:26:02.611725 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:26:02.611710    4477 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/738731d9-e739-4c3e-8936-9c4d01f5fc04-maas-api-tls podName:738731d9-e739-4c3e-8936-9c4d01f5fc04 nodeName:}" failed. No retries permitted until 2026-10-30 22:26:03.111690726 +0000 UTC m=+1147.037616914 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/738731d9-e739-4c3e-8936-9c4d01f5fc04-maas-api-tls") pod "maas-api-e2e-shared-a-w0-5983ea-586646597c-z222v" (UID: "738731d9-e739-4c3e-8936-9c4d01f5fc04") : secret "maas-api-e2e-shared-a-w0-5983ea-serving-cert" not found
Oct 30 22:26:02.611772 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:26:02.611740    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/738731d9-e739-4c3e-8936-9c4d01f5fc04-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-5983ea-586646597c-z222v\" (UID: \"738731d9-e739-4c3e-8936-9c4d01f5fc04\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5983ea-586646597c-z222v"
Oct 30 22:26:02.613668 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:26:02.613643    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/738731d9-e739-4c3e-8936-9c4d01f5fc04-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-5983ea-586646597c-z222v\" (UID: \"738731d9-e739-4c3e-8936-9c4d01f5fc04\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5983ea-586646597c-z222v"
Oct 30 22:26:02.638815 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:26:02.638624    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58lwp\" (UniqueName: \"kubernetes.io/projected/738731d9-e739-4c3e-8936-9c4d01f5fc04-kube-api-access-58lwp\") pod \"maas-api-e2e-shared-a-w0-5983ea-586646597c-z222v\" (UID: \"738731d9-e739-4c3e-8936-9c4d01f5fc04\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5983ea-586646597c-z222v"
Oct 30 22:26:03.118806 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:26:03.118767    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/738731d9-e739-4c3e-8936-9c4d01f5fc04-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-5983ea-586646597c-z222v\" (UID: \"738731d9-e739-4c3e-8936-9c4d01f5fc04\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5983ea-586646597c-z222v"
Oct 30 22:26:03.121179 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:26:03.121149    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/738731d9-e739-4c3e-8936-9c4d01f5fc04-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-5983ea-586646597c-z222v\" (UID: \"738731d9-e739-4c3e-8936-9c4d01f5fc04\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5983ea-586646597c-z222v"
Oct 30 22:26:03.362947 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:26:03.362912    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5983ea-586646597c-z222v"
Oct 30 22:26:03.676344 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:26:03.676315    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5983ea-586646597c-z222v"]
Oct 30 22:26:03.677883 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:26:03.677793    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod738731d9_e739_4c3e_8936_9c4d01f5fc04.slice/crio-04d6d433fd6f668a0dc12e4808e7e1ffcf58d2b5b82662b5865bda2d897fd925 WatchSource:0}: Error finding container 04d6d433fd6f668a0dc12e4808e7e1ffcf58d2b5b82662b5865bda2d897fd925: Status 404 returned error can't find the container with id 04d6d433fd6f668a0dc12e4808e7e1ffcf58d2b5b82662b5865bda2d897fd925
Oct 30 22:26:03.946795 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:26:03.946717    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5983ea-586646597c-z222v" event={"ID":"738731d9-e739-4c3e-8936-9c4d01f5fc04","Type":"ContainerStarted","Data":"04d6d433fd6f668a0dc12e4808e7e1ffcf58d2b5b82662b5865bda2d897fd925"}
Oct 30 22:26:04.955214 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:26:04.955182    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5983ea-586646597c-z222v" event={"ID":"738731d9-e739-4c3e-8936-9c4d01f5fc04","Type":"ContainerStarted","Data":"af0ac40b526d5b6b4fa4a597572ff9e9d205c384db85782f0828ff46bd2c43b7"}
Oct 30 22:26:04.955671 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:26:04.955257    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5983ea-586646597c-z222v"
Oct 30 22:26:04.996222 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:26:04.996171    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5983ea-586646597c-z222v" podStartSLOduration=2.6704889510000003 podStartE2EDuration="2.996158329s" totalImagesPullingTime="325.669378ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:26:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:26:04.992236634 +0000 UTC m=+1148.918162860" watchObservedRunningTime="2026-10-30 22:26:04.996158329 +0000 UTC m=+1148.922084534"
Oct 30 22:26:10.970952 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:26:10.970924    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5983ea-586646597c-z222v"
Oct 30 22:26:22.906333 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:26:22.906298    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-2949c143/e2e-lifecycle-model-2949c143-kserve-74f9489d85-g9btm"]
Oct 30 22:26:22.911214 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:26:22.911191    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-2949c143/e2e-lifecycle-model-2949c143-kserve-74f9489d85-g9btm"
Oct 30 22:26:22.931650 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:26:22.931625    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-2949c143/e2e-lifecycle-model-2949c143-kserve-74f9489d85-g9btm"]
Oct 30 22:26:22.933703 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:26:22.933677    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-2949c143\"/\"default-dockercfg-l8bk4\""
Oct 30 22:26:22.935405 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:26:22.935223    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-2949c143\"/\"e2e-lifecycle-model-2949c143-kserve-self-signed-certs\""
Oct 30 22:26:23.016419 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:26:23.016380    4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-2949c143\"/\"openshift-service-ca.crt\""
Oct 30 22:26:23.016591 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:26:23.016454    4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-2949c143\"/\"kube-root-ca.crt\""
Oct 30 22:26:23.032547 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:26:23.032499    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0883afa-5295-4347-848e-ebfd7f831844-home\") pod \"e2e-lifecycle-model-2949c143-kserve-74f9489d85-g9btm\" (UID: \"c0883afa-5295-4347-848e-ebfd7f831844\") " pod="ai-tenant-e2e-mt-2949c143/e2e-lifecycle-model-2949c143-kserve-74f9489d85-g9btm"
Oct 30 22:26:23.032675 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:26:23.032576    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0883afa-5295-4347-848e-ebfd7f831844-tmp-dir\") pod \"e2e-lifecycle-model-2949c143-kserve-74f9489d85-g9btm\" (UID: \"c0883afa-5295-4347-848e-ebfd7f831844\") " pod="ai-tenant-e2e-mt-2949c143/e2e-lifecycle-model-2949c143-kserve-74f9489d85-g9btm"
Oct 30 22:26:23.032751 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:26:23.032672    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52xbc\" (UniqueName: \"kubernetes.io/projected/c0883afa-5295-4347-848e-ebfd7f831844-kube-api-access-52xbc\") pod \"e2e-lifecycle-model-2949c143-kserve-74f9489d85-g9btm\" (UID: \"c0883afa-5295-4347-848e-ebfd7f831844\") " pod="ai-tenant-e2e-mt-2949c143/e2e-lifecycle-model-2949c143-kserve-74f9489d85-g9btm"
Oct 30 22:26:23.032850 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:26:23.032828    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0883afa-5295-4347-848e-ebfd7f831844-model-cache\") pod \"e2e-lifecycle-model-2949c143-kserve-74f9489d85-g9btm\" (UID: \"c0883afa-5295-4347-848e-ebfd7f831844\") " pod="ai-tenant-e2e-mt-2949c143/e2e-lifecycle-model-2949c143-kserve-74f9489d85-g9btm"
Oct 30 22:26:23.032925 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:26:23.032911    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0883afa-5295-4347-848e-ebfd7f831844-dshm\") pod \"e2e-lifecycle-model-2949c143-kserve-74f9489d85-g9btm\" (UID: \"c0883afa-5295-4347-848e-ebfd7f831844\") " pod="ai-tenant-e2e-mt-2949c143/e2e-lifecycle-model-2949c143-kserve-74f9489d85-g9btm"
Oct 30 22:26:23.032985 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:26:23.032958    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0883afa-5295-4347-848e-ebfd7f831844-tls-certs\") pod \"e2e-lifecycle-model-2949c143-kserve-74f9489d85-g9btm\" (UID: \"c0883afa-5295-4347-848e-ebfd7f831844\") " pod="ai-tenant-e2e-mt-2949c143/e2e-lifecycle-model-2949c143-kserve-74f9489d85-g9btm"
Oct 30 22:26:23.134287 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:26:23.134241    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0883afa-5295-4347-848e-ebfd7f831844-home\") pod \"e2e-lifecycle-model-2949c143-kserve-74f9489d85-g9btm\" (UID: \"c0883afa-5295-4347-848e-ebfd7f831844\") " pod="ai-tenant-e2e-mt-2949c143/e2e-lifecycle-model-2949c143-kserve-74f9489d85-g9btm"
Oct 30 22:26:23.134504 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:26:23.134311    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0883afa-5295-4347-848e-ebfd7f831844-tmp-dir\") pod \"e2e-lifecycle-model-2949c143-kserve-74f9489d85-g9btm\" (UID: \"c0883afa-5295-4347-848e-ebfd7f831844\") " pod="ai-tenant-e2e-mt-2949c143/e2e-lifecycle-model-2949c143-kserve-74f9489d85-g9btm"
Oct 30 22:26:23.134504 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:26:23.134385    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52xbc\" (UniqueName: \"kubernetes.io/projected/c0883afa-5295-4347-848e-ebfd7f831844-kube-api-access-52xbc\") pod \"e2e-lifecycle-model-2949c143-kserve-74f9489d85-g9btm\" (UID: \"c0883afa-5295-4347-848e-ebfd7f831844\") " pod="ai-tenant-e2e-mt-2949c143/e2e-lifecycle-model-2949c143-kserve-74f9489d85-g9btm"
Oct 30 22:26:23.134639 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:26:23.134515    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0883afa-5295-4347-848e-ebfd7f831844-model-cache\") pod \"e2e-lifecycle-model-2949c143-kserve-74f9489d85-g9btm\" (UID: \"c0883afa-5295-4347-848e-ebfd7f831844\") " pod="ai-tenant-e2e-mt-2949c143/e2e-lifecycle-model-2949c143-kserve-74f9489d85-g9btm"
Oct 30 22:26:23.134639 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:26:23.134593    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0883afa-5295-4347-848e-ebfd7f831844-dshm\") pod \"e2e-lifecycle-model-2949c143-kserve-74f9489d85-g9btm\" (UID: \"c0883afa-5295-4347-848e-ebfd7f831844\") " pod="ai-tenant-e2e-mt-2949c143/e2e-lifecycle-model-2949c143-kserve-74f9489d85-g9btm"
Oct 30 22:26:23.134639 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:26:23.134622    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0883afa-5295-4347-848e-ebfd7f831844-tls-certs\") pod \"e2e-lifecycle-model-2949c143-kserve-74f9489d85-g9btm\" (UID: \"c0883afa-5295-4347-848e-ebfd7f831844\") " pod="ai-tenant-e2e-mt-2949c143/e2e-lifecycle-model-2949c143-kserve-74f9489d85-g9btm"
Oct 30 22:26:23.134829 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:26:23.134679    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0883afa-5295-4347-848e-ebfd7f831844-home\") pod \"e2e-lifecycle-model-2949c143-kserve-74f9489d85-g9btm\" (UID: \"c0883afa-5295-4347-848e-ebfd7f831844\") " pod="ai-tenant-e2e-mt-2949c143/e2e-lifecycle-model-2949c143-kserve-74f9489d85-g9btm"
Oct 30 22:26:23.135373 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:26:23.135350    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0883afa-5295-4347-848e-ebfd7f831844-tmp-dir\") pod \"e2e-lifecycle-model-2949c143-kserve-74f9489d85-g9btm\" (UID: \"c0883afa-5295-4347-848e-ebfd7f831844\") " pod="ai-tenant-e2e-mt-2949c143/e2e-lifecycle-model-2949c143-kserve-74f9489d85-g9btm"
Oct 30 22:26:23.137326 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:26:23.137301    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0883afa-5295-4347-848e-ebfd7f831844-model-cache\") pod \"e2e-lifecycle-model-2949c143-kserve-74f9489d85-g9btm\" (UID: \"c0883afa-5295-4347-848e-ebfd7f831844\") " pod="ai-tenant-e2e-mt-2949c143/e2e-lifecycle-model-2949c143-kserve-74f9489d85-g9btm"
Oct 30 22:26:23.137854 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:26:23.137831    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0883afa-5295-4347-848e-ebfd7f831844-tls-certs\") pod \"e2e-lifecycle-model-2949c143-kserve-74f9489d85-g9btm\" (UID: \"c0883afa-5295-4347-848e-ebfd7f831844\") " pod="ai-tenant-e2e-mt-2949c143/e2e-lifecycle-model-2949c143-kserve-74f9489d85-g9btm"
Oct 30 22:26:23.138133 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:26:23.138035    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0883afa-5295-4347-848e-ebfd7f831844-dshm\") pod \"e2e-lifecycle-model-2949c143-kserve-74f9489d85-g9btm\" (UID: \"c0883afa-5295-4347-848e-ebfd7f831844\") " pod="ai-tenant-e2e-mt-2949c143/e2e-lifecycle-model-2949c143-kserve-74f9489d85-g9btm"
Oct 30 22:26:23.154438 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:26:23.154408    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52xbc\" (UniqueName: \"kubernetes.io/projected/c0883afa-5295-4347-848e-ebfd7f831844-kube-api-access-52xbc\") pod \"e2e-lifecycle-model-2949c143-kserve-74f9489d85-g9btm\" (UID: \"c0883afa-5295-4347-848e-ebfd7f831844\") " pod="ai-tenant-e2e-mt-2949c143/e2e-lifecycle-model-2949c143-kserve-74f9489d85-g9btm"
Oct 30 22:26:23.234308 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:26:23.234160    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-2949c143/e2e-lifecycle-model-2949c143-kserve-74f9489d85-g9btm"
Oct 30 22:26:23.558444 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:26:23.558411    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-2949c143/e2e-lifecycle-model-2949c143-kserve-74f9489d85-g9btm"]
Oct 30 22:26:23.561903 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:26:23.561874    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc0883afa_5295_4347_848e_ebfd7f831844.slice/crio-30a3ddf7b4dbffccbec81afde71bdc8a47ef580b0c2e99bcfdef67403cfad566 WatchSource:0}: Error finding container 30a3ddf7b4dbffccbec81afde71bdc8a47ef580b0c2e99bcfdef67403cfad566: Status 404 returned error can't find the container with id 30a3ddf7b4dbffccbec81afde71bdc8a47ef580b0c2e99bcfdef67403cfad566
Oct 30 22:26:24.092215 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:26:24.092177    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-2949c143/e2e-lifecycle-model-2949c143-kserve-74f9489d85-g9btm" event={"ID":"c0883afa-5295-4347-848e-ebfd7f831844","Type":"ContainerStarted","Data":"0d95da8c6195f34e4d0fa8605e44efb3426ea2caab7a7e491126743f9edb41e9"}
Oct 30 22:26:24.092677 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:26:24.092223    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-2949c143/e2e-lifecycle-model-2949c143-kserve-74f9489d85-g9btm" event={"ID":"c0883afa-5295-4347-848e-ebfd7f831844","Type":"ContainerStarted","Data":"30a3ddf7b4dbffccbec81afde71bdc8a47ef580b0c2e99bcfdef67403cfad566"}
Oct 30 22:26:24.129972 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:26:24.129913    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-2949c143/e2e-lifecycle-model-2949c143-kserve-74f9489d85-g9btm" podStartSLOduration=2.129897817 podStartE2EDuration="2.129897817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:26:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:26:24.127919077 +0000 UTC m=+1168.053845284" watchObservedRunningTime="2026-10-30 22:26:24.129897817 +0000 UTC m=+1168.055824025"
Oct 30 22:26:24.235112 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:26:24.235081    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-2949c143/e2e-lifecycle-model-2949c143-kserve-74f9489d85-g9btm"
Oct 30 22:26:26.266394 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:26:26.266361    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 22:26:26.270143 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:26:26.270036    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6bz92"]
Oct 30 22:26:33.234703 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:26:33.234670    4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-2949c143/e2e-lifecycle-model-2949c143-kserve-74f9489d85-g9btm"
Oct 30 22:26:33.247237 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:26:33.247213    4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-2949c143/e2e-lifecycle-model-2949c143-kserve-74f9489d85-g9btm"
Oct 30 22:26:34.175695 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:26:34.175666    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-2949c143/e2e-lifecycle-model-2949c143-kserve-74f9489d85-g9btm"
Oct 30 22:26:45.607427 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:26:45.607393    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3cd77f-75497bfcfc-6sdhc"]
Oct 30 22:26:45.611580 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:26:45.611563    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3cd77f-75497bfcfc-6sdhc"
Oct 30 22:26:45.627652 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:26:45.627635    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-3cd77f-serving-cert\""
Oct 30 22:26:45.632600 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:26:45.632578    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3cd77f-75497bfcfc-6sdhc"]
Oct 30 22:26:45.789943 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:26:45.789905    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjvk9\" (UniqueName: \"kubernetes.io/projected/0ec852a7-f59e-4475-817a-fb57174a901d-kube-api-access-qjvk9\") pod \"maas-api-e2e-shared-b-w2-3cd77f-75497bfcfc-6sdhc\" (UID: \"0ec852a7-f59e-4475-817a-fb57174a901d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3cd77f-75497bfcfc-6sdhc"
Oct 30 22:26:45.790161 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:26:45.790076    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ec852a7-f59e-4475-817a-fb57174a901d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-3cd77f-75497bfcfc-6sdhc\" (UID: \"0ec852a7-f59e-4475-817a-fb57174a901d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3cd77f-75497bfcfc-6sdhc"
Oct 30 22:26:45.790161 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:26:45.790137    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ec852a7-f59e-4475-817a-fb57174a901d-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-3cd77f-75497bfcfc-6sdhc\" (UID: \"0ec852a7-f59e-4475-817a-fb57174a901d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3cd77f-75497bfcfc-6sdhc"
Oct 30 22:26:45.790337 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:26:45.790315    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ec852a7-f59e-4475-817a-fb57174a901d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-3cd77f-75497bfcfc-6sdhc\" (UID: \"0ec852a7-f59e-4475-817a-fb57174a901d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3cd77f-75497bfcfc-6sdhc"
Oct 30 22:26:45.891909 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:26:45.891875    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ec852a7-f59e-4475-817a-fb57174a901d-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-3cd77f-75497bfcfc-6sdhc\" (UID: \"0ec852a7-f59e-4475-817a-fb57174a901d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3cd77f-75497bfcfc-6sdhc"
Oct 30 22:26:45.892106 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:26:45.892072    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ec852a7-f59e-4475-817a-fb57174a901d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-3cd77f-75497bfcfc-6sdhc\" (UID: \"0ec852a7-f59e-4475-817a-fb57174a901d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3cd77f-75497bfcfc-6sdhc"
Oct 30 22:26:45.892171 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:26:45.892149    4477 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3cd77f-serving-cert: secret "maas-api-e2e-shared-b-w2-3cd77f-serving-cert" not found
Oct 30 22:26:45.892212 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:26:45.892198    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjvk9\" (UniqueName: \"kubernetes.io/projected/0ec852a7-f59e-4475-817a-fb57174a901d-kube-api-access-qjvk9\") pod \"maas-api-e2e-shared-b-w2-3cd77f-75497bfcfc-6sdhc\" (UID: \"0ec852a7-f59e-4475-817a-fb57174a901d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3cd77f-75497bfcfc-6sdhc"
Oct 30 22:26:45.892246 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:26:45.892219    4477 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ec852a7-f59e-4475-817a-fb57174a901d-maas-api-tls podName:0ec852a7-f59e-4475-817a-fb57174a901d nodeName:}" failed. No retries permitted until 2026-10-30 22:26:46.392197278 +0000 UTC m=+1190.318123474 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0ec852a7-f59e-4475-817a-fb57174a901d-maas-api-tls") pod "maas-api-e2e-shared-b-w2-3cd77f-75497bfcfc-6sdhc" (UID: "0ec852a7-f59e-4475-817a-fb57174a901d") : secret "maas-api-e2e-shared-b-w2-3cd77f-serving-cert" not found
Oct 30 22:26:45.892379 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:26:45.892361    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ec852a7-f59e-4475-817a-fb57174a901d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-3cd77f-75497bfcfc-6sdhc\" (UID: \"0ec852a7-f59e-4475-817a-fb57174a901d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3cd77f-75497bfcfc-6sdhc"
Oct 30 22:26:45.892969 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:26:45.892943    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ec852a7-f59e-4475-817a-fb57174a901d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-3cd77f-75497bfcfc-6sdhc\" (UID: \"0ec852a7-f59e-4475-817a-fb57174a901d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3cd77f-75497bfcfc-6sdhc"
Oct 30 22:26:45.894316 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:26:45.894298    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ec852a7-f59e-4475-817a-fb57174a901d-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-3cd77f-75497bfcfc-6sdhc\" (UID: \"0ec852a7-f59e-4475-817a-fb57174a901d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3cd77f-75497bfcfc-6sdhc"
Oct 30 22:26:45.909929 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:26:45.909904    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjvk9\" (UniqueName: \"kubernetes.io/projected/0ec852a7-f59e-4475-817a-fb57174a901d-kube-api-access-qjvk9\") pod \"maas-api-e2e-shared-b-w2-3cd77f-75497bfcfc-6sdhc\" (UID: \"0ec852a7-f59e-4475-817a-fb57174a901d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3cd77f-75497bfcfc-6sdhc"
Oct 30 22:26:46.400513 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:26:46.400484    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ec852a7-f59e-4475-817a-fb57174a901d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-3cd77f-75497bfcfc-6sdhc\" (UID: \"0ec852a7-f59e-4475-817a-fb57174a901d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3cd77f-75497bfcfc-6sdhc"
Oct 30 22:26:46.402846 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:26:46.402825    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ec852a7-f59e-4475-817a-fb57174a901d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-3cd77f-75497bfcfc-6sdhc\" (UID: \"0ec852a7-f59e-4475-817a-fb57174a901d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3cd77f-75497bfcfc-6sdhc"
Oct 30 22:26:46.526207 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:26:46.526180    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3cd77f-75497bfcfc-6sdhc"
Oct 30 22:26:46.829673 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:26:46.829638    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3cd77f-75497bfcfc-6sdhc"]
Oct 30 22:26:46.830314 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:26:46.830285    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ec852a7_f59e_4475_817a_fb57174a901d.slice/crio-4234b978d68784b107ea46428831ae863de84b8c05e1ed84d258b1ea5ead9c8c WatchSource:0}: Error finding container 4234b978d68784b107ea46428831ae863de84b8c05e1ed84d258b1ea5ead9c8c: Status 404 returned error can't find the container with id 4234b978d68784b107ea46428831ae863de84b8c05e1ed84d258b1ea5ead9c8c
Oct 30 22:26:47.269101 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:26:47.268156    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3cd77f-75497bfcfc-6sdhc" event={"ID":"0ec852a7-f59e-4475-817a-fb57174a901d","Type":"ContainerStarted","Data":"a75ca9b111ba4ef20bde65bf1888b689cf49aff6449ae82964a780ed70bf741d"}
Oct 30 22:26:47.269101 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:26:47.268213    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3cd77f-75497bfcfc-6sdhc" event={"ID":"0ec852a7-f59e-4475-817a-fb57174a901d","Type":"ContainerStarted","Data":"4234b978d68784b107ea46428831ae863de84b8c05e1ed84d258b1ea5ead9c8c"}
Oct 30 22:26:47.269101 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:26:47.268331    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3cd77f-75497bfcfc-6sdhc"
Oct 30 22:26:47.310848 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:26:47.310792    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3cd77f-75497bfcfc-6sdhc" podStartSLOduration=1.978464885 podStartE2EDuration="2.310777062s" totalImagesPullingTime="332.312177ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:26:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:26:47.307005778 +0000 UTC m=+1191.232931985" watchObservedRunningTime="2026-10-30 22:26:47.310777062 +0000 UTC m=+1191.236703268"
Oct 30 22:26:53.279099 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:26:53.279071    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3cd77f-75497bfcfc-6sdhc"
Oct 30 22:26:56.273461 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:26:56.273411    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 22:26:56.755969 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:26:56.755945    4477 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jcf8l_015f218d-bf1f-4e7e-bb5d-b805af003246/ovn-acl-logging/0.log"
Oct 30 22:26:56.820894 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:26:56.820866    4477 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jcf8l_015f218d-bf1f-4e7e-bb5d-b805af003246/ovn-acl-logging/0.log"
Oct 30 22:26:57.545383 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:26:57.545353    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6af86e-57878656ff-qv65k"]
Oct 30 22:26:57.549975 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:26:57.549949    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6af86e-57878656ff-qv65k"
Oct 30 22:26:57.574136 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:26:57.574101    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-6af86e-serving-cert\""
Oct 30 22:26:57.578176 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:26:57.578151    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6af86e-57878656ff-qv65k"]
Oct 30 22:26:57.671331 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:26:57.671298    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/763c695b-9f27-48e4-80a6-ae3c22bc9816-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-6af86e-57878656ff-qv65k\" (UID: \"763c695b-9f27-48e4-80a6-ae3c22bc9816\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6af86e-57878656ff-qv65k"
Oct 30 22:26:57.671463 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:26:57.671355    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/763c695b-9f27-48e4-80a6-ae3c22bc9816-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-6af86e-57878656ff-qv65k\" (UID: \"763c695b-9f27-48e4-80a6-ae3c22bc9816\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6af86e-57878656ff-qv65k"
Oct 30 22:26:57.671558 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:26:57.671485    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/763c695b-9f27-48e4-80a6-ae3c22bc9816-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-6af86e-57878656ff-qv65k\" (UID: \"763c695b-9f27-48e4-80a6-ae3c22bc9816\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6af86e-57878656ff-qv65k"
Oct 30 22:26:57.671558 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:26:57.671543    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qp2pm\" (UniqueName: \"kubernetes.io/projected/763c695b-9f27-48e4-80a6-ae3c22bc9816-kube-api-access-qp2pm\") pod \"maas-api-e2e-shared-b-w0-6af86e-57878656ff-qv65k\" (UID: \"763c695b-9f27-48e4-80a6-ae3c22bc9816\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6af86e-57878656ff-qv65k"
Oct 30 22:26:57.773955 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:26:57.773913    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/763c695b-9f27-48e4-80a6-ae3c22bc9816-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-6af86e-57878656ff-qv65k\" (UID: \"763c695b-9f27-48e4-80a6-ae3c22bc9816\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6af86e-57878656ff-qv65k"
Oct 30 22:26:57.774253 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:26:57.774223    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/763c695b-9f27-48e4-80a6-ae3c22bc9816-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-6af86e-57878656ff-qv65k\" (UID: \"763c695b-9f27-48e4-80a6-ae3c22bc9816\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6af86e-57878656ff-qv65k"
Oct 30 22:26:57.774387 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:26:57.774343    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/763c695b-9f27-48e4-80a6-ae3c22bc9816-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-6af86e-57878656ff-qv65k\" (UID: \"763c695b-9f27-48e4-80a6-ae3c22bc9816\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6af86e-57878656ff-qv65k"
Oct 30 22:26:57.774461 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:26:57.774396    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qp2pm\" (UniqueName: \"kubernetes.io/projected/763c695b-9f27-48e4-80a6-ae3c22bc9816-kube-api-access-qp2pm\") pod \"maas-api-e2e-shared-b-w0-6af86e-57878656ff-qv65k\" (UID: \"763c695b-9f27-48e4-80a6-ae3c22bc9816\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6af86e-57878656ff-qv65k"
Oct 30 22:26:57.774864 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:26:57.774835    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/763c695b-9f27-48e4-80a6-ae3c22bc9816-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-6af86e-57878656ff-qv65k\" (UID: \"763c695b-9f27-48e4-80a6-ae3c22bc9816\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6af86e-57878656ff-qv65k"
Oct 30 22:26:57.778092 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:26:57.778028    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/763c695b-9f27-48e4-80a6-ae3c22bc9816-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-6af86e-57878656ff-qv65k\" (UID: \"763c695b-9f27-48e4-80a6-ae3c22bc9816\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6af86e-57878656ff-qv65k"
Oct 30 22:26:57.790342 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:26:57.790271    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/763c695b-9f27-48e4-80a6-ae3c22bc9816-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-6af86e-57878656ff-qv65k\" (UID: \"763c695b-9f27-48e4-80a6-ae3c22bc9816\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6af86e-57878656ff-qv65k"
Oct 30 22:26:57.804852 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:26:57.797788    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qp2pm\" (UniqueName: \"kubernetes.io/projected/763c695b-9f27-48e4-80a6-ae3c22bc9816-kube-api-access-qp2pm\") pod \"maas-api-e2e-shared-b-w0-6af86e-57878656ff-qv65k\" (UID: \"763c695b-9f27-48e4-80a6-ae3c22bc9816\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6af86e-57878656ff-qv65k"
Oct 30 22:26:57.864119 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:26:57.864089    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6af86e-57878656ff-qv65k"
Oct 30 22:26:58.183647 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:26:58.183548    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6af86e-57878656ff-qv65k"]
Oct 30 22:26:58.186892 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:26:58.186858    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod763c695b_9f27_48e4_80a6_ae3c22bc9816.slice/crio-600eedb45dd6bed6ad95a553f79e0f063bc1a7d14c40dc659d8a2912a8c7847d WatchSource:0}: Error finding container 600eedb45dd6bed6ad95a553f79e0f063bc1a7d14c40dc659d8a2912a8c7847d: Status 404 returned error can't find the container with id 600eedb45dd6bed6ad95a553f79e0f063bc1a7d14c40dc659d8a2912a8c7847d
Oct 30 22:26:58.350684 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:26:58.350644    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6af86e-57878656ff-qv65k" event={"ID":"763c695b-9f27-48e4-80a6-ae3c22bc9816","Type":"ContainerStarted","Data":"600eedb45dd6bed6ad95a553f79e0f063bc1a7d14c40dc659d8a2912a8c7847d"}
Oct 30 22:26:59.358799 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:26:59.358768    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6af86e-57878656ff-qv65k" event={"ID":"763c695b-9f27-48e4-80a6-ae3c22bc9816","Type":"ContainerStarted","Data":"0c0de0b66292e3ad1c60c817316d1f69abd7006fa228e14233610401b441faf4"}
Oct 30 22:26:59.359247 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:26:59.358856    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6af86e-57878656ff-qv65k"
Oct 30 22:26:59.396362 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:26:59.396320    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6af86e-57878656ff-qv65k" podStartSLOduration=2.016938457 podStartE2EDuration="2.396308152s" totalImagesPullingTime="379.369695ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:26:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:26:59.39494637 +0000 UTC m=+1203.320872618" watchObservedRunningTime="2026-10-30 22:26:59.396308152 +0000 UTC m=+1203.322234358"
Oct 30 22:27:05.370117 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:27:05.370088    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6af86e-57878656ff-qv65k"
Oct 30 22:27:26.266717 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:27:26.266675    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 22:27:28.875125 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:27:28.875094    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6bz92"]
Oct 30 22:27:37.030408 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:27:37.030352    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74c3a2d4-fqxxl"]
Oct 30 22:27:37.039354 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:27:37.039332    4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74c3a2d4-fqxxl"]
Oct 30 22:27:37.314952 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:27:37.314882    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d399fb07-78b4b78864-6x646"]
Oct 30 22:27:37.320349 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:27:37.320328    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d399fb07-78b4b78864-6x646"
Oct 30 22:27:37.340790 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:27:37.340763    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d399fb07-78b4b78864-6x646"]
Oct 30 22:27:37.346653 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:27:37.346618    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-d399fb07-serving-cert\""
Oct 30 22:27:37.459030 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:27:37.459001    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f11bbf1-c90c-4e20-a603-93dffec97774-openshift-service-ca\") pod \"maas-api-e2e-mt-d399fb07-78b4b78864-6x646\" (UID: \"8f11bbf1-c90c-4e20-a603-93dffec97774\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d399fb07-78b4b78864-6x646"
Oct 30 22:27:37.459172 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:27:37.459137    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4css8\" (UniqueName: \"kubernetes.io/projected/8f11bbf1-c90c-4e20-a603-93dffec97774-kube-api-access-4css8\") pod \"maas-api-e2e-mt-d399fb07-78b4b78864-6x646\" (UID: \"8f11bbf1-c90c-4e20-a603-93dffec97774\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d399fb07-78b4b78864-6x646"
Oct 30 22:27:37.459214 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:27:37.459188    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f11bbf1-c90c-4e20-a603-93dffec97774-maas-api-tls\") pod \"maas-api-e2e-mt-d399fb07-78b4b78864-6x646\" (UID: \"8f11bbf1-c90c-4e20-a603-93dffec97774\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d399fb07-78b4b78864-6x646"
Oct 30 22:27:37.459313 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:27:37.459295    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f11bbf1-c90c-4e20-a603-93dffec97774-metrics-certs\") pod \"maas-api-e2e-mt-d399fb07-78b4b78864-6x646\" (UID: \"8f11bbf1-c90c-4e20-a603-93dffec97774\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d399fb07-78b4b78864-6x646"
Oct 30 22:27:37.560727 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:27:37.560697    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f11bbf1-c90c-4e20-a603-93dffec97774-openshift-service-ca\") pod \"maas-api-e2e-mt-d399fb07-78b4b78864-6x646\" (UID: \"8f11bbf1-c90c-4e20-a603-93dffec97774\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d399fb07-78b4b78864-6x646"
Oct 30 22:27:37.560967 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:27:37.560823    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4css8\" (UniqueName: \"kubernetes.io/projected/8f11bbf1-c90c-4e20-a603-93dffec97774-kube-api-access-4css8\") pod \"maas-api-e2e-mt-d399fb07-78b4b78864-6x646\" (UID: \"8f11bbf1-c90c-4e20-a603-93dffec97774\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d399fb07-78b4b78864-6x646"
Oct 30 22:27:37.560967 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:27:37.560870    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f11bbf1-c90c-4e20-a603-93dffec97774-maas-api-tls\") pod \"maas-api-e2e-mt-d399fb07-78b4b78864-6x646\" (UID: \"8f11bbf1-c90c-4e20-a603-93dffec97774\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d399fb07-78b4b78864-6x646"
Oct 30 22:27:37.560967 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:27:37.560953    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f11bbf1-c90c-4e20-a603-93dffec97774-metrics-certs\") pod \"maas-api-e2e-mt-d399fb07-78b4b78864-6x646\" (UID: \"8f11bbf1-c90c-4e20-a603-93dffec97774\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d399fb07-78b4b78864-6x646"
Oct 30 22:27:37.562356 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:27:37.561360    4477 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-d399fb07-serving-cert: secret "maas-api-e2e-mt-d399fb07-serving-cert" not found
Oct 30 22:27:37.562356 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:27:37.561469    4477 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f11bbf1-c90c-4e20-a603-93dffec97774-maas-api-tls podName:8f11bbf1-c90c-4e20-a603-93dffec97774 nodeName:}" failed. No retries permitted until 2026-10-30 22:27:38.06144807 +0000 UTC m=+1241.987374266 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8f11bbf1-c90c-4e20-a603-93dffec97774-maas-api-tls") pod "maas-api-e2e-mt-d399fb07-78b4b78864-6x646" (UID: "8f11bbf1-c90c-4e20-a603-93dffec97774") : secret "maas-api-e2e-mt-d399fb07-serving-cert" not found
Oct 30 22:27:37.562356 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:27:37.561495    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f11bbf1-c90c-4e20-a603-93dffec97774-openshift-service-ca\") pod \"maas-api-e2e-mt-d399fb07-78b4b78864-6x646\" (UID: \"8f11bbf1-c90c-4e20-a603-93dffec97774\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d399fb07-78b4b78864-6x646"
Oct 30 22:27:37.563652 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:27:37.563632    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f11bbf1-c90c-4e20-a603-93dffec97774-metrics-certs\") pod \"maas-api-e2e-mt-d399fb07-78b4b78864-6x646\" (UID: \"8f11bbf1-c90c-4e20-a603-93dffec97774\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d399fb07-78b4b78864-6x646"
Oct 30 22:27:37.578698 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:27:37.578628    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4css8\" (UniqueName: \"kubernetes.io/projected/8f11bbf1-c90c-4e20-a603-93dffec97774-kube-api-access-4css8\") pod \"maas-api-e2e-mt-d399fb07-78b4b78864-6x646\" (UID: \"8f11bbf1-c90c-4e20-a603-93dffec97774\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d399fb07-78b4b78864-6x646"
Oct 30 22:27:38.067815 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:27:38.067782    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f11bbf1-c90c-4e20-a603-93dffec97774-maas-api-tls\") pod \"maas-api-e2e-mt-d399fb07-78b4b78864-6x646\" (UID: \"8f11bbf1-c90c-4e20-a603-93dffec97774\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d399fb07-78b4b78864-6x646"
Oct 30 22:27:38.070274 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:27:38.070243    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f11bbf1-c90c-4e20-a603-93dffec97774-maas-api-tls\") pod \"maas-api-e2e-mt-d399fb07-78b4b78864-6x646\" (UID: \"8f11bbf1-c90c-4e20-a603-93dffec97774\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d399fb07-78b4b78864-6x646"
Oct 30 22:27:38.194926 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:27:38.194894    4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25689f97-ea82-4662-976a-a4e8f7f2282b" path="/var/lib/kubelet/pods/25689f97-ea82-4662-976a-a4e8f7f2282b/volumes"
Oct 30 22:27:38.236307 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:27:38.236275    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d399fb07-78b4b78864-6x646"
Oct 30 22:27:38.588910 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:27:38.588877    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d399fb07-78b4b78864-6x646"]
Oct 30 22:27:38.698066 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:27:38.697266    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d399fb07-78b4b78864-6x646" event={"ID":"8f11bbf1-c90c-4e20-a603-93dffec97774","Type":"ContainerStarted","Data":"f8a46bbb5f1a6b39ad0cdb273fe0ebc1e4612c96e79dfbc97c4441678598e902"}
Oct 30 22:27:39.709087 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:27:39.709032    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d399fb07-78b4b78864-6x646" event={"ID":"8f11bbf1-c90c-4e20-a603-93dffec97774","Type":"ContainerStarted","Data":"aa382678738ec7916cfa5fb07c528de9a1737c4df0249bd734583c8a7fdd4247"}
Oct 30 22:27:39.709537 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:27:39.709280    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d399fb07-78b4b78864-6x646"
Oct 30 22:27:39.747653 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:27:39.747530    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d399fb07-78b4b78864-6x646" podStartSLOduration=2.356217643 podStartE2EDuration="2.747513485s" totalImagesPullingTime="391.295842ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:27:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:27:39.745872679 +0000 UTC m=+1243.671798900" watchObservedRunningTime="2026-10-30 22:27:39.747513485 +0000 UTC m=+1243.673439693"
Oct 30 22:27:41.218289 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:27:41.218252    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-6f2eba-787bf86545-5pm9l"]
Oct 30 22:27:41.223259 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:27:41.223235    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-6f2eba-787bf86545-5pm9l"
Oct 30 22:27:41.250130 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:27:41.250094    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w5-6f2eba-serving-cert\""
Oct 30 22:27:41.255929 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:27:41.255899    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-6f2eba-787bf86545-5pm9l"]
Oct 30 22:27:41.317604 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:27:41.317574    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d8ff9bb-aa7c-46f4-8ff8-d85ba96870a4-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-6f2eba-787bf86545-5pm9l\" (UID: \"2d8ff9bb-aa7c-46f4-8ff8-d85ba96870a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-6f2eba-787bf86545-5pm9l"
Oct 30 22:27:41.317757 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:27:41.317617    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d8ff9bb-aa7c-46f4-8ff8-d85ba96870a4-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-6f2eba-787bf86545-5pm9l\" (UID: \"2d8ff9bb-aa7c-46f4-8ff8-d85ba96870a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-6f2eba-787bf86545-5pm9l"
Oct 30 22:27:41.317802 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:27:41.317767    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cclv7\" (UniqueName: \"kubernetes.io/projected/2d8ff9bb-aa7c-46f4-8ff8-d85ba96870a4-kube-api-access-cclv7\") pod \"maas-api-e2e-shared-a-w5-6f2eba-787bf86545-5pm9l\" (UID: \"2d8ff9bb-aa7c-46f4-8ff8-d85ba96870a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-6f2eba-787bf86545-5pm9l"
Oct 30 22:27:41.317985 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:27:41.317963    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d8ff9bb-aa7c-46f4-8ff8-d85ba96870a4-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-6f2eba-787bf86545-5pm9l\" (UID: \"2d8ff9bb-aa7c-46f4-8ff8-d85ba96870a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-6f2eba-787bf86545-5pm9l"
Oct 30 22:27:41.419254 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:27:41.419210    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d8ff9bb-aa7c-46f4-8ff8-d85ba96870a4-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-6f2eba-787bf86545-5pm9l\" (UID: \"2d8ff9bb-aa7c-46f4-8ff8-d85ba96870a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-6f2eba-787bf86545-5pm9l"
Oct 30 22:27:41.419254 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:27:41.419256    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d8ff9bb-aa7c-46f4-8ff8-d85ba96870a4-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-6f2eba-787bf86545-5pm9l\" (UID: \"2d8ff9bb-aa7c-46f4-8ff8-d85ba96870a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-6f2eba-787bf86545-5pm9l"
Oct 30 22:27:41.419767 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:27:41.419369    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cclv7\" (UniqueName: \"kubernetes.io/projected/2d8ff9bb-aa7c-46f4-8ff8-d85ba96870a4-kube-api-access-cclv7\") pod \"maas-api-e2e-shared-a-w5-6f2eba-787bf86545-5pm9l\" (UID: \"2d8ff9bb-aa7c-46f4-8ff8-d85ba96870a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-6f2eba-787bf86545-5pm9l"
Oct 30 22:27:41.419767 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:27:41.419607    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d8ff9bb-aa7c-46f4-8ff8-d85ba96870a4-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-6f2eba-787bf86545-5pm9l\" (UID: \"2d8ff9bb-aa7c-46f4-8ff8-d85ba96870a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-6f2eba-787bf86545-5pm9l"
Oct 30 22:27:41.420293 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:27:41.420263    4477 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-6f2eba-serving-cert: secret "maas-api-e2e-shared-a-w5-6f2eba-serving-cert" not found
Oct 30 22:27:41.420419 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:27:41.420337    4477 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d8ff9bb-aa7c-46f4-8ff8-d85ba96870a4-maas-api-tls podName:2d8ff9bb-aa7c-46f4-8ff8-d85ba96870a4 nodeName:}" failed. No retries permitted until 2026-10-30 22:27:41.920318129 +0000 UTC m=+1245.846244313 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2d8ff9bb-aa7c-46f4-8ff8-d85ba96870a4-maas-api-tls") pod "maas-api-e2e-shared-a-w5-6f2eba-787bf86545-5pm9l" (UID: "2d8ff9bb-aa7c-46f4-8ff8-d85ba96870a4") : secret "maas-api-e2e-shared-a-w5-6f2eba-serving-cert" not found
Oct 30 22:27:41.420761 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:27:41.420736    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d8ff9bb-aa7c-46f4-8ff8-d85ba96870a4-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-6f2eba-787bf86545-5pm9l\" (UID: \"2d8ff9bb-aa7c-46f4-8ff8-d85ba96870a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-6f2eba-787bf86545-5pm9l"
Oct 30 22:27:41.429987 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:27:41.424065    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d8ff9bb-aa7c-46f4-8ff8-d85ba96870a4-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-6f2eba-787bf86545-5pm9l\" (UID: \"2d8ff9bb-aa7c-46f4-8ff8-d85ba96870a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-6f2eba-787bf86545-5pm9l"
Oct 30 22:27:41.441605 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:27:41.441574    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cclv7\" (UniqueName: \"kubernetes.io/projected/2d8ff9bb-aa7c-46f4-8ff8-d85ba96870a4-kube-api-access-cclv7\") pod \"maas-api-e2e-shared-a-w5-6f2eba-787bf86545-5pm9l\" (UID: \"2d8ff9bb-aa7c-46f4-8ff8-d85ba96870a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-6f2eba-787bf86545-5pm9l"
Oct 30 22:27:41.926344 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:27:41.926314    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d8ff9bb-aa7c-46f4-8ff8-d85ba96870a4-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-6f2eba-787bf86545-5pm9l\" (UID: \"2d8ff9bb-aa7c-46f4-8ff8-d85ba96870a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-6f2eba-787bf86545-5pm9l"
Oct 30 22:27:41.928796 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:27:41.928768    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d8ff9bb-aa7c-46f4-8ff8-d85ba96870a4-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-6f2eba-787bf86545-5pm9l\" (UID: \"2d8ff9bb-aa7c-46f4-8ff8-d85ba96870a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-6f2eba-787bf86545-5pm9l"
Oct 30 22:27:42.138002 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:27:42.137954    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-6f2eba-787bf86545-5pm9l"
Oct 30 22:27:42.542958 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:27:42.542931    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-6f2eba-787bf86545-5pm9l"]
Oct 30 22:27:42.548394 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:27:42.548341    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d8ff9bb_aa7c_46f4_8ff8_d85ba96870a4.slice/crio-69e7e31cbd87bbb6cd131e599708fdd8c4dce153a2da67db9542706080b4e591 WatchSource:0}: Error finding container 69e7e31cbd87bbb6cd131e599708fdd8c4dce153a2da67db9542706080b4e591: Status 404 returned error can't find the container with id 69e7e31cbd87bbb6cd131e599708fdd8c4dce153a2da67db9542706080b4e591
Oct 30 22:27:42.736619 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:27:42.736584    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-6f2eba-787bf86545-5pm9l" event={"ID":"2d8ff9bb-aa7c-46f4-8ff8-d85ba96870a4","Type":"ContainerStarted","Data":"69e7e31cbd87bbb6cd131e599708fdd8c4dce153a2da67db9542706080b4e591"}
Oct 30 22:27:43.745813 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:27:43.745778    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-6f2eba-787bf86545-5pm9l" event={"ID":"2d8ff9bb-aa7c-46f4-8ff8-d85ba96870a4","Type":"ContainerStarted","Data":"d508ab4a260bbe48ff27d505bdba7309118828242a01bed72c6b54aad6f84376"}
Oct 30 22:27:43.746286 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:27:43.745979    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-6f2eba-787bf86545-5pm9l"
Oct 30 22:27:43.785148 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:27:43.785089    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-6f2eba-787bf86545-5pm9l" podStartSLOduration=2.501490246 podStartE2EDuration="2.7850718s" totalImagesPullingTime="283.581554ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:27:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:27:43.781552131 +0000 UTC m=+1247.707478337" watchObservedRunningTime="2026-10-30 22:27:43.7850718 +0000 UTC m=+1247.710998008"
Oct 30 22:27:45.721611 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:27:45.721584    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d399fb07-78b4b78864-6x646"
Oct 30 22:27:46.034212 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:27:46.034144    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cad8417b-6lnzr"]
Oct 30 22:27:46.037675 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:27:46.037652    4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cad8417b-6lnzr"]
Oct 30 22:27:46.194895 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:27:46.194860    4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7efed674-a0ee-438e-aa0c-664a46b49c6b" path="/var/lib/kubelet/pods/7efed674-a0ee-438e-aa0c-664a46b49c6b/volumes"
Oct 30 22:27:49.758353 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:27:49.758323    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-6f2eba-787bf86545-5pm9l"
Oct 30 22:27:56.267631 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:27:56.267591    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 22:27:58.043730 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:27:58.043700    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f9e4062-hww42"]
Oct 30 22:27:58.054834 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:27:58.054808    4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f9e4062-hww42"]
Oct 30 22:27:58.196025 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:27:58.195992    4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3f10f78-1693-47a0-9f53-eadb243ca6d8" path="/var/lib/kubelet/pods/f3f10f78-1693-47a0-9f53-eadb243ca6d8/volumes"
Oct 30 22:28:10.045583 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:28:10.045548    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-220e38c3-7cx7g"]
Oct 30 22:28:10.056898 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:28:10.056866    4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-220e38c3-7cx7g"]
Oct 30 22:28:10.206373 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:28:10.206338    4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae8f9b4b-3744-40f2-a157-6ccb78a0151f" path="/var/lib/kubelet/pods/ae8f9b4b-3744-40f2-a157-6ccb78a0151f/volumes"
Oct 30 22:28:21.627771 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:28:21.627738    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-47qsn"]
Oct 30 22:28:21.632424 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:28:21.632401    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-47qsn"
Oct 30 22:28:21.656965 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:28:21.656922    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-d399fb07-openshift-default-dockercfg-gkn9c\""
Oct 30 22:28:21.659147 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:28:21.659122    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-47qsn"]
Oct 30 22:28:21.734015 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:28:21.733980    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m45v4\" (UniqueName: \"kubernetes.io/projected/a620edfb-2b72-4c1f-8b51-9a13d2cfb485-kube-api-access-m45v4\") pod \"e2e-mt-d399fb07-openshift-default-6d54788969-47qsn\" (UID: \"a620edfb-2b72-4c1f-8b51-9a13d2cfb485\") " pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-47qsn"
Oct 30 22:28:21.734208 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:28:21.734087    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a620edfb-2b72-4c1f-8b51-9a13d2cfb485-istio-data\") pod \"e2e-mt-d399fb07-openshift-default-6d54788969-47qsn\" (UID: \"a620edfb-2b72-4c1f-8b51-9a13d2cfb485\") " pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-47qsn"
Oct 30 22:28:21.734263 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:28:21.734223    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a620edfb-2b72-4c1f-8b51-9a13d2cfb485-workload-certs\") pod \"e2e-mt-d399fb07-openshift-default-6d54788969-47qsn\" (UID: \"a620edfb-2b72-4c1f-8b51-9a13d2cfb485\") " pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-47qsn"
Oct 30 22:28:21.734320 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:28:21.734267    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a620edfb-2b72-4c1f-8b51-9a13d2cfb485-credential-socket\") pod \"e2e-mt-d399fb07-openshift-default-6d54788969-47qsn\" (UID: \"a620edfb-2b72-4c1f-8b51-9a13d2cfb485\") " pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-47qsn"
Oct 30 22:28:21.734376 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:28:21.734355    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a620edfb-2b72-4c1f-8b51-9a13d2cfb485-istio-podinfo\") pod \"e2e-mt-d399fb07-openshift-default-6d54788969-47qsn\" (UID: \"a620edfb-2b72-4c1f-8b51-9a13d2cfb485\") " pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-47qsn"
Oct 30 22:28:21.734431 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:28:21.734383    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a620edfb-2b72-4c1f-8b51-9a13d2cfb485-istio-token\") pod \"e2e-mt-d399fb07-openshift-default-6d54788969-47qsn\" (UID: \"a620edfb-2b72-4c1f-8b51-9a13d2cfb485\") " pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-47qsn"
Oct 30 22:28:21.734593 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:28:21.734568    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a620edfb-2b72-4c1f-8b51-9a13d2cfb485-istiod-ca-cert\") pod \"e2e-mt-d399fb07-openshift-default-6d54788969-47qsn\" (UID: \"a620edfb-2b72-4c1f-8b51-9a13d2cfb485\") " pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-47qsn"
Oct 30 22:28:21.734777 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:28:21.734640    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a620edfb-2b72-4c1f-8b51-9a13d2cfb485-workload-socket\") pod \"e2e-mt-d399fb07-openshift-default-6d54788969-47qsn\" (UID: \"a620edfb-2b72-4c1f-8b51-9a13d2cfb485\") " pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-47qsn"
Oct 30 22:28:21.734777 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:28:21.734667    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a620edfb-2b72-4c1f-8b51-9a13d2cfb485-istio-envoy\") pod \"e2e-mt-d399fb07-openshift-default-6d54788969-47qsn\" (UID: \"a620edfb-2b72-4c1f-8b51-9a13d2cfb485\") " pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-47qsn"
Oct 30 22:28:21.835788 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:28:21.835759    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m45v4\" (UniqueName: \"kubernetes.io/projected/a620edfb-2b72-4c1f-8b51-9a13d2cfb485-kube-api-access-m45v4\") pod \"e2e-mt-d399fb07-openshift-default-6d54788969-47qsn\" (UID: \"a620edfb-2b72-4c1f-8b51-9a13d2cfb485\") " pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-47qsn"
Oct 30 22:28:21.835926 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:28:21.835807    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a620edfb-2b72-4c1f-8b51-9a13d2cfb485-istio-data\") pod \"e2e-mt-d399fb07-openshift-default-6d54788969-47qsn\" (UID: \"a620edfb-2b72-4c1f-8b51-9a13d2cfb485\") " pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-47qsn"
Oct 30 22:28:21.835926 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:28:21.835826    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a620edfb-2b72-4c1f-8b51-9a13d2cfb485-workload-certs\") pod \"e2e-mt-d399fb07-openshift-default-6d54788969-47qsn\" (UID: \"a620edfb-2b72-4c1f-8b51-9a13d2cfb485\") " pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-47qsn"
Oct 30 22:28:21.836034 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:28:21.835932    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a620edfb-2b72-4c1f-8b51-9a13d2cfb485-credential-socket\") pod \"e2e-mt-d399fb07-openshift-default-6d54788969-47qsn\" (UID: \"a620edfb-2b72-4c1f-8b51-9a13d2cfb485\") " pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-47qsn"
Oct 30 22:28:21.836034 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:28:21.835981    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a620edfb-2b72-4c1f-8b51-9a13d2cfb485-istio-podinfo\") pod \"e2e-mt-d399fb07-openshift-default-6d54788969-47qsn\" (UID: \"a620edfb-2b72-4c1f-8b51-9a13d2cfb485\") " pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-47qsn"
Oct 30 22:28:21.836034 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:28:21.836004    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a620edfb-2b72-4c1f-8b51-9a13d2cfb485-istio-token\") pod \"e2e-mt-d399fb07-openshift-default-6d54788969-47qsn\" (UID: \"a620edfb-2b72-4c1f-8b51-9a13d2cfb485\") " pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-47qsn"
Oct 30 22:28:21.836208 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:28:21.836156    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a620edfb-2b72-4c1f-8b51-9a13d2cfb485-istiod-ca-cert\") pod \"e2e-mt-d399fb07-openshift-default-6d54788969-47qsn\" (UID: \"a620edfb-2b72-4c1f-8b51-9a13d2cfb485\") " pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-47qsn"
Oct 30 22:28:21.836208 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:28:21.836165    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a620edfb-2b72-4c1f-8b51-9a13d2cfb485-workload-certs\") pod \"e2e-mt-d399fb07-openshift-default-6d54788969-47qsn\" (UID: \"a620edfb-2b72-4c1f-8b51-9a13d2cfb485\") " pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-47qsn"
Oct 30 22:28:21.836321 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:28:21.836252    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a620edfb-2b72-4c1f-8b51-9a13d2cfb485-istio-data\") pod \"e2e-mt-d399fb07-openshift-default-6d54788969-47qsn\" (UID: \"a620edfb-2b72-4c1f-8b51-9a13d2cfb485\") " pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-47qsn"
Oct 30 22:28:21.836321 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:28:21.836274    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a620edfb-2b72-4c1f-8b51-9a13d2cfb485-credential-socket\") pod \"e2e-mt-d399fb07-openshift-default-6d54788969-47qsn\" (UID: \"a620edfb-2b72-4c1f-8b51-9a13d2cfb485\") " pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-47qsn"
Oct 30 22:28:21.836321 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:28:21.836258    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a620edfb-2b72-4c1f-8b51-9a13d2cfb485-workload-socket\") pod \"e2e-mt-d399fb07-openshift-default-6d54788969-47qsn\" (UID: \"a620edfb-2b72-4c1f-8b51-9a13d2cfb485\") " pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-47qsn"
Oct 30 22:28:21.836479 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:28:21.836321    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a620edfb-2b72-4c1f-8b51-9a13d2cfb485-istio-envoy\") pod \"e2e-mt-d399fb07-openshift-default-6d54788969-47qsn\" (UID: \"a620edfb-2b72-4c1f-8b51-9a13d2cfb485\") " pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-47qsn"
Oct 30 22:28:21.836550 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:28:21.836523    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a620edfb-2b72-4c1f-8b51-9a13d2cfb485-workload-socket\") pod \"e2e-mt-d399fb07-openshift-default-6d54788969-47qsn\" (UID: \"a620edfb-2b72-4c1f-8b51-9a13d2cfb485\") " pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-47qsn"
Oct 30 22:28:21.836802 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:28:21.836785    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a620edfb-2b72-4c1f-8b51-9a13d2cfb485-istiod-ca-cert\") pod \"e2e-mt-d399fb07-openshift-default-6d54788969-47qsn\" (UID: \"a620edfb-2b72-4c1f-8b51-9a13d2cfb485\") " pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-47qsn"
Oct 30 22:28:21.838407 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:28:21.838383    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a620edfb-2b72-4c1f-8b51-9a13d2cfb485-istio-envoy\") pod \"e2e-mt-d399fb07-openshift-default-6d54788969-47qsn\" (UID: \"a620edfb-2b72-4c1f-8b51-9a13d2cfb485\") " pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-47qsn"
Oct 30 22:28:21.838600 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:28:21.838578    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a620edfb-2b72-4c1f-8b51-9a13d2cfb485-istio-podinfo\") pod \"e2e-mt-d399fb07-openshift-default-6d54788969-47qsn\" (UID: \"a620edfb-2b72-4c1f-8b51-9a13d2cfb485\") " pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-47qsn"
Oct 30 22:28:21.854029 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:28:21.854000    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a620edfb-2b72-4c1f-8b51-9a13d2cfb485-istio-token\") pod \"e2e-mt-d399fb07-openshift-default-6d54788969-47qsn\" (UID: \"a620edfb-2b72-4c1f-8b51-9a13d2cfb485\") " pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-47qsn"
Oct 30 22:28:21.854205 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:28:21.854186    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m45v4\" (UniqueName: \"kubernetes.io/projected/a620edfb-2b72-4c1f-8b51-9a13d2cfb485-kube-api-access-m45v4\") pod \"e2e-mt-d399fb07-openshift-default-6d54788969-47qsn\" (UID: \"a620edfb-2b72-4c1f-8b51-9a13d2cfb485\") " pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-47qsn"
Oct 30 22:28:21.952738 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:28:21.952402    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-47qsn"
Oct 30 22:28:22.409106 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:28:22.409077    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-47qsn"]
Oct 30 22:28:22.411223 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:28:22.411195    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda620edfb_2b72_4c1f_8b51_9a13d2cfb485.slice/crio-73175e746e718c26c13580fa71f1c299bff826a3df6f111658079e8a49ef22d8 WatchSource:0}: Error finding container 73175e746e718c26c13580fa71f1c299bff826a3df6f111658079e8a49ef22d8: Status 404 returned error can't find the container with id 73175e746e718c26c13580fa71f1c299bff826a3df6f111658079e8a49ef22d8
Oct 30 22:28:22.414096 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:28:22.414056    4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:28:22.414183 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:28:22.414131    4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:28:22.414183 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:28:22.414162    4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:28:23.131323 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:28:23.131284    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-47qsn" event={"ID":"a620edfb-2b72-4c1f-8b51-9a13d2cfb485","Type":"ContainerStarted","Data":"601e77bb3bfbdfa9e4c4e7edd54bac35079d6bccbd64b290c5aee78520417df3"}
Oct 30 22:28:23.131808 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:28:23.131354    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-47qsn" event={"ID":"a620edfb-2b72-4c1f-8b51-9a13d2cfb485","Type":"ContainerStarted","Data":"73175e746e718c26c13580fa71f1c299bff826a3df6f111658079e8a49ef22d8"}
Oct 30 22:28:23.180533 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:28:23.180484    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-47qsn" podStartSLOduration=2.180470601 podStartE2EDuration="2.180470601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:28:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:28:23.179318442 +0000 UTC m=+1287.105244670" watchObservedRunningTime="2026-10-30 22:28:23.180470601 +0000 UTC m=+1287.106396806"
Oct 30 22:28:23.953181 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:28:23.953115    4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-47qsn"
Oct 30 22:28:23.954645 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:28:23.954609    4477 patch_prober.go:29] interesting pod/e2e-mt-d399fb07-openshift-default-6d54788969-47qsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Oct 30 22:28:23.954781 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:28:23.954693    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-47qsn" podUID="a620edfb-2b72-4c1f-8b51-9a13d2cfb485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Oct 30 22:28:24.953825 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:28:24.953789    4477 patch_prober.go:29] interesting pod/e2e-mt-d399fb07-openshift-default-6d54788969-47qsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Oct 30 22:28:24.954259 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:28:24.953853    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-47qsn" podUID="a620edfb-2b72-4c1f-8b51-9a13d2cfb485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Oct 30 22:28:25.953366 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:28:25.953333    4477 patch_prober.go:29] interesting pod/e2e-mt-d399fb07-openshift-default-6d54788969-47qsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Oct 30 22:28:25.953510 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:28:25.953394    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-47qsn" podUID="a620edfb-2b72-4c1f-8b51-9a13d2cfb485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Oct 30 22:28:26.268398 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:28:26.268325    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 22:28:26.953603 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:28:26.953568    4477 patch_prober.go:29] interesting pod/e2e-mt-d399fb07-openshift-default-6d54788969-47qsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Oct 30 22:28:26.953773 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:28:26.953627    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-47qsn" podUID="a620edfb-2b72-4c1f-8b51-9a13d2cfb485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Oct 30 22:28:27.953412 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:28:27.953377    4477 patch_prober.go:29] interesting pod/e2e-mt-d399fb07-openshift-default-6d54788969-47qsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Oct 30 22:28:27.954248 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:28:27.953437    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-47qsn" podUID="a620edfb-2b72-4c1f-8b51-9a13d2cfb485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Oct 30 22:28:28.957855 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:28:28.957826    4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-47qsn"
Oct 30 22:28:28.958323 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:28:28.958304    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-47qsn"
Oct 30 22:28:28.959347 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:28:28.959330    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-47qsn"
Oct 30 22:28:33.773285 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:28:33.773249    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6bz92"]
Oct 30 22:28:46.640861 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:28:46.640779    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-510f80-5475f9dfb-njms4"]
Oct 30 22:28:46.645342 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:28:46.645320    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-510f80-5475f9dfb-njms4"
Oct 30 22:28:46.658588 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:28:46.658567    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w5-510f80-serving-cert\""
Oct 30 22:28:46.672619 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:28:46.672590    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-510f80-5475f9dfb-njms4"]
Oct 30 22:28:46.723318 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:28:46.723292    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00783ce5-3f4a-4649-9c93-28b8a1c01b1e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-510f80-5475f9dfb-njms4\" (UID: \"00783ce5-3f4a-4649-9c93-28b8a1c01b1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-510f80-5475f9dfb-njms4"
Oct 30 22:28:46.723451 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:28:46.723352    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00783ce5-3f4a-4649-9c93-28b8a1c01b1e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-510f80-5475f9dfb-njms4\" (UID: \"00783ce5-3f4a-4649-9c93-28b8a1c01b1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-510f80-5475f9dfb-njms4"
Oct 30 22:28:46.723497 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:28:46.723450    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00783ce5-3f4a-4649-9c93-28b8a1c01b1e-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-510f80-5475f9dfb-njms4\" (UID: \"00783ce5-3f4a-4649-9c93-28b8a1c01b1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-510f80-5475f9dfb-njms4"
Oct 30 22:28:46.723634 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:28:46.723612    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sv86w\" (UniqueName: \"kubernetes.io/projected/00783ce5-3f4a-4649-9c93-28b8a1c01b1e-kube-api-access-sv86w\") pod \"maas-api-e2e-shared-b-w5-510f80-5475f9dfb-njms4\" (UID: \"00783ce5-3f4a-4649-9c93-28b8a1c01b1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-510f80-5475f9dfb-njms4"
Oct 30 22:28:46.824757 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:28:46.824729    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00783ce5-3f4a-4649-9c93-28b8a1c01b1e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-510f80-5475f9dfb-njms4\" (UID: \"00783ce5-3f4a-4649-9c93-28b8a1c01b1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-510f80-5475f9dfb-njms4"
Oct 30 22:28:46.824896 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:28:46.824794    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00783ce5-3f4a-4649-9c93-28b8a1c01b1e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-510f80-5475f9dfb-njms4\" (UID: \"00783ce5-3f4a-4649-9c93-28b8a1c01b1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-510f80-5475f9dfb-njms4"
Oct 30 22:28:46.824896 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:28:46.824838    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00783ce5-3f4a-4649-9c93-28b8a1c01b1e-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-510f80-5475f9dfb-njms4\" (UID: \"00783ce5-3f4a-4649-9c93-28b8a1c01b1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-510f80-5475f9dfb-njms4"
Oct 30 22:28:46.824997 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:28:46.824980    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sv86w\" (UniqueName: \"kubernetes.io/projected/00783ce5-3f4a-4649-9c93-28b8a1c01b1e-kube-api-access-sv86w\") pod \"maas-api-e2e-shared-b-w5-510f80-5475f9dfb-njms4\" (UID: \"00783ce5-3f4a-4649-9c93-28b8a1c01b1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-510f80-5475f9dfb-njms4"
Oct 30 22:28:46.825503 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:28:46.825481    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00783ce5-3f4a-4649-9c93-28b8a1c01b1e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-510f80-5475f9dfb-njms4\" (UID: \"00783ce5-3f4a-4649-9c93-28b8a1c01b1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-510f80-5475f9dfb-njms4"
Oct 30 22:28:46.827398 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:28:46.827375    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00783ce5-3f4a-4649-9c93-28b8a1c01b1e-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-510f80-5475f9dfb-njms4\" (UID: \"00783ce5-3f4a-4649-9c93-28b8a1c01b1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-510f80-5475f9dfb-njms4"
Oct 30 22:28:46.827483 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:28:46.827396    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00783ce5-3f4a-4649-9c93-28b8a1c01b1e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-510f80-5475f9dfb-njms4\" (UID: \"00783ce5-3f4a-4649-9c93-28b8a1c01b1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-510f80-5475f9dfb-njms4"
Oct 30 22:28:46.842677 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:28:46.842654    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sv86w\" (UniqueName: \"kubernetes.io/projected/00783ce5-3f4a-4649-9c93-28b8a1c01b1e-kube-api-access-sv86w\") pod \"maas-api-e2e-shared-b-w5-510f80-5475f9dfb-njms4\" (UID: \"00783ce5-3f4a-4649-9c93-28b8a1c01b1e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-510f80-5475f9dfb-njms4"
Oct 30 22:28:46.959610 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:28:46.959517    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-510f80-5475f9dfb-njms4"
Oct 30 22:28:47.274744 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:28:47.274692    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-510f80-5475f9dfb-njms4"]
Oct 30 22:28:47.277812 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:28:47.277778    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00783ce5_3f4a_4649_9c93_28b8a1c01b1e.slice/crio-5e8e3a0b01e62f4b0fcc77be3f7d7c845f40b18f6c8ac52a041073c638e9a0fe WatchSource:0}: Error finding container 5e8e3a0b01e62f4b0fcc77be3f7d7c845f40b18f6c8ac52a041073c638e9a0fe: Status 404 returned error can't find the container with id 5e8e3a0b01e62f4b0fcc77be3f7d7c845f40b18f6c8ac52a041073c638e9a0fe
Oct 30 22:28:47.279205 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:28:47.279183    4477 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 22:28:47.319679 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:28:47.319654    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-510f80-5475f9dfb-njms4" event={"ID":"00783ce5-3f4a-4649-9c93-28b8a1c01b1e","Type":"ContainerStarted","Data":"5e8e3a0b01e62f4b0fcc77be3f7d7c845f40b18f6c8ac52a041073c638e9a0fe"}
Oct 30 22:28:48.328998 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:28:48.328966    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-510f80-5475f9dfb-njms4" event={"ID":"00783ce5-3f4a-4649-9c93-28b8a1c01b1e","Type":"ContainerStarted","Data":"fe5de093d371f6396f53000157af70d6ccde6be912b369ffbd4bef41896ab151"}
Oct 30 22:28:48.329414 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:28:48.329175    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-510f80-5475f9dfb-njms4"
Oct 30 22:28:48.366683 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:28:48.366637    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-510f80-5475f9dfb-njms4" podStartSLOduration=2.065811084 podStartE2EDuration="2.366623247s" totalImagesPullingTime="300.812163ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:28:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:28:48.365942299 +0000 UTC m=+1312.291868506" watchObservedRunningTime="2026-10-30 22:28:48.366623247 +0000 UTC m=+1312.292549452"
Oct 30 22:28:54.342014 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:28:54.341981    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-510f80-5475f9dfb-njms4"
Oct 30 22:28:56.268384 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:28:56.268345    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 22:28:56.622827 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:28:56.622756    4477 scope.go:122] "RemoveContainer" containerID="ec41613219cad5a5a3fd5fea9cf2c651869b1e5e32296d652f02b0e3c291dcc9"
Oct 30 22:28:56.632444 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:28:56.632420    4477 scope.go:122] "RemoveContainer" containerID="6287fed66e05f45cc47318eff344fbfb86b5674ddfd1a2a2cf5f3875d8c4f7b9"
Oct 30 22:28:56.641107 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:28:56.641086    4477 scope.go:122] "RemoveContainer" containerID="235ceba0e4b32f16deec3aa16da4fc194e289058686bee89f6a99ba7340e56b5"
Oct 30 22:29:26.267538 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:29:26.267501    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 22:29:28.622303 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:29:28.622267    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2949c143-5d858999bf-bwqg9"]
Oct 30 22:29:28.622771 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:29:28.622558    4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2949c143-5d858999bf-bwqg9" podUID="c16202d5-34f4-4112-9972-b95ebbd47068" containerName="maas-api" containerID="cri-o://82342d31edcfc2dbd1df9dfb3f960ed1720e19f92fbd1a283abf5096e8793511" gracePeriod=30
Oct 30 22:29:29.008545 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:29:29.008513    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2949c143-5d858999bf-bwqg9"
Oct 30 22:29:29.076881 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:29:29.074667    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c16202d5-34f4-4112-9972-b95ebbd47068-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c16202d5-34f4-4112-9972-b95ebbd47068-openshift-service-ca\") pod \"c16202d5-34f4-4112-9972-b95ebbd47068\" (UID: \"c16202d5-34f4-4112-9972-b95ebbd47068\") "
Oct 30 22:29:29.076881 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:29:29.074810    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c16202d5-34f4-4112-9972-b95ebbd47068-kube-api-access-hjwlq\" (UniqueName: \"kubernetes.io/projected/c16202d5-34f4-4112-9972-b95ebbd47068-kube-api-access-hjwlq\") pod \"c16202d5-34f4-4112-9972-b95ebbd47068\" (UID: \"c16202d5-34f4-4112-9972-b95ebbd47068\") "
Oct 30 22:29:29.076881 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:29:29.074976    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c16202d5-34f4-4112-9972-b95ebbd47068-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c16202d5-34f4-4112-9972-b95ebbd47068-maas-api-tls\") pod \"c16202d5-34f4-4112-9972-b95ebbd47068\" (UID: \"c16202d5-34f4-4112-9972-b95ebbd47068\") "
Oct 30 22:29:29.076881 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:29:29.075019    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c16202d5-34f4-4112-9972-b95ebbd47068-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c16202d5-34f4-4112-9972-b95ebbd47068-metrics-certs\") pod \"c16202d5-34f4-4112-9972-b95ebbd47068\" (UID: \"c16202d5-34f4-4112-9972-b95ebbd47068\") "
Oct 30 22:29:29.081713 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:29:29.081683    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c16202d5-34f4-4112-9972-b95ebbd47068-openshift-service-ca" pod "c16202d5-34f4-4112-9972-b95ebbd47068" (UID: "c16202d5-34f4-4112-9972-b95ebbd47068"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:29:29.091183 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:29:29.086555    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c16202d5-34f4-4112-9972-b95ebbd47068-kube-api-access-hjwlq" pod "c16202d5-34f4-4112-9972-b95ebbd47068" (UID: "c16202d5-34f4-4112-9972-b95ebbd47068"). InnerVolumeSpecName "kube-api-access-hjwlq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:29:29.091183 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:29:29.086884    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c16202d5-34f4-4112-9972-b95ebbd47068-metrics-certs" pod "c16202d5-34f4-4112-9972-b95ebbd47068" (UID: "c16202d5-34f4-4112-9972-b95ebbd47068"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:29:29.091183 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:29:29.087241    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c16202d5-34f4-4112-9972-b95ebbd47068-maas-api-tls" pod "c16202d5-34f4-4112-9972-b95ebbd47068" (UID: "c16202d5-34f4-4112-9972-b95ebbd47068"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:29:29.185116 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:29:29.182400    4477 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c16202d5-34f4-4112-9972-b95ebbd47068-openshift-service-ca\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:29:29.185116 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:29:29.182439    4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjwlq\" (UniqueName: \"kubernetes.io/projected/c16202d5-34f4-4112-9972-b95ebbd47068-kube-api-access-hjwlq\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:29:29.185116 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:29:29.182454    4477 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c16202d5-34f4-4112-9972-b95ebbd47068-maas-api-tls\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:29:29.185116 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:29:29.182468    4477 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c16202d5-34f4-4112-9972-b95ebbd47068-metrics-certs\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:29:29.678303 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:29:29.678268    4477 generic.go:356] "Generic (PLEG): container finished" podID="c16202d5-34f4-4112-9972-b95ebbd47068" containerID="82342d31edcfc2dbd1df9dfb3f960ed1720e19f92fbd1a283abf5096e8793511" exitCode=0
Oct 30 22:29:29.678800 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:29:29.678418    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2949c143-5d858999bf-bwqg9" event={"ID":"c16202d5-34f4-4112-9972-b95ebbd47068","Type":"ContainerDied","Data":"82342d31edcfc2dbd1df9dfb3f960ed1720e19f92fbd1a283abf5096e8793511"}
Oct 30 22:29:29.678800 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:29:29.678458    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2949c143-5d858999bf-bwqg9" event={"ID":"c16202d5-34f4-4112-9972-b95ebbd47068","Type":"ContainerDied","Data":"677baacd575395375a0a758468bf64846e12051c14b259bc71240e6c694c890d"}
Oct 30 22:29:29.678800 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:29:29.678485    4477 scope.go:122] "RemoveContainer" containerID="82342d31edcfc2dbd1df9dfb3f960ed1720e19f92fbd1a283abf5096e8793511"
Oct 30 22:29:29.678800 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:29:29.678642    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2949c143-5d858999bf-bwqg9"
Oct 30 22:29:29.698998 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:29:29.698953    4477 scope.go:122] "RemoveContainer" containerID="82342d31edcfc2dbd1df9dfb3f960ed1720e19f92fbd1a283abf5096e8793511"
Oct 30 22:29:29.699906 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:29:29.699878    4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82342d31edcfc2dbd1df9dfb3f960ed1720e19f92fbd1a283abf5096e8793511\": container with ID starting with 82342d31edcfc2dbd1df9dfb3f960ed1720e19f92fbd1a283abf5096e8793511 not found: ID does not exist" containerID="82342d31edcfc2dbd1df9dfb3f960ed1720e19f92fbd1a283abf5096e8793511"
Oct 30 22:29:29.700020 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:29:29.699933    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82342d31edcfc2dbd1df9dfb3f960ed1720e19f92fbd1a283abf5096e8793511"} err="failed to get container status \"82342d31edcfc2dbd1df9dfb3f960ed1720e19f92fbd1a283abf5096e8793511\": rpc error: code = NotFound desc = could not find container \"82342d31edcfc2dbd1df9dfb3f960ed1720e19f92fbd1a283abf5096e8793511\": container with ID starting with 82342d31edcfc2dbd1df9dfb3f960ed1720e19f92fbd1a283abf5096e8793511 not found: ID does not exist"
Oct 30 22:29:29.731332 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:29:29.731305    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2949c143-5d858999bf-bwqg9"]
Oct 30 22:29:29.741461 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:29:29.741437    4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2949c143-5d858999bf-bwqg9"]
Oct 30 22:29:30.195799 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:29:30.195767    4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c16202d5-34f4-4112-9972-b95ebbd47068" path="/var/lib/kubelet/pods/c16202d5-34f4-4112-9972-b95ebbd47068/volumes"
Oct 30 22:29:43.679064 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:29:43.675030    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6bz92"]
Oct 30 22:29:56.268359 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:29:56.268321    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 22:29:56.674779 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:29:56.674720    4477 scope.go:122] "RemoveContainer" containerID="00f829ab61cf84bbb84b023f08995516af72f3787df1d7f4370ea62282ac3ec8"
Oct 30 22:30:00.201412 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:00.201383    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3cd77f-29889990-tj449"]
Oct 30 22:30:00.202342 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:00.202318    4477 cpu_manager.go:414] "containerMap: removing container" podUID="c16202d5-34f4-4112-9972-b95ebbd47068" containerName="maas-api"
Oct 30 22:30:00.202342 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:00.202340    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c16202d5-34f4-4112-9972-b95ebbd47068" containerName="maas-api"
Oct 30 22:30:00.202541 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:00.202492    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="c16202d5-34f4-4112-9972-b95ebbd47068" containerName="maas-api"
Oct 30 22:30:00.208053 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:00.208016    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3cd77f-29889990-tj449"
Oct 30 22:30:00.208741 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:00.208711    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2b1e88-29889990-mtmwc"]
Oct 30 22:30:00.256672 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:00.256633    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3cd77f-29889990-tj449"]
Oct 30 22:30:00.256860 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:00.256685    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2b1e88-29889990-mtmwc"]
Oct 30 22:30:00.256860 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:00.256759    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2b1e88-29889990-mtmwc"
Oct 30 22:30:00.294855 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:00.294826    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 22:30:00.336389 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:00.336357    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/852aabae-2489-4afe-ae9e-20d551cbf9a0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-2b1e88-29889990-mtmwc\" (UID: \"852aabae-2489-4afe-ae9e-20d551cbf9a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2b1e88-29889990-mtmwc"
Oct 30 22:30:00.336567 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:00.336544    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea5c4363-22dd-446e-a6df-1a0ce6497a87-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-3cd77f-29889990-tj449\" (UID: \"ea5c4363-22dd-446e-a6df-1a0ce6497a87\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3cd77f-29889990-tj449"
Oct 30 22:30:00.336635 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:00.336618    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bq2qp\" (UniqueName: \"kubernetes.io/projected/ea5c4363-22dd-446e-a6df-1a0ce6497a87-kube-api-access-bq2qp\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-3cd77f-29889990-tj449\" (UID: \"ea5c4363-22dd-446e-a6df-1a0ce6497a87\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3cd77f-29889990-tj449"
Oct 30 22:30:00.336776 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:00.336753    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72dtm\" (UniqueName: \"kubernetes.io/projected/852aabae-2489-4afe-ae9e-20d551cbf9a0-kube-api-access-72dtm\") pod \"maas-api-key-cleanup-e2e-worker-w1-2b1e88-29889990-mtmwc\" (UID: \"852aabae-2489-4afe-ae9e-20d551cbf9a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2b1e88-29889990-mtmwc"
Oct 30 22:30:00.438491 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:00.438458    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72dtm\" (UniqueName: \"kubernetes.io/projected/852aabae-2489-4afe-ae9e-20d551cbf9a0-kube-api-access-72dtm\") pod \"maas-api-key-cleanup-e2e-worker-w1-2b1e88-29889990-mtmwc\" (UID: \"852aabae-2489-4afe-ae9e-20d551cbf9a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2b1e88-29889990-mtmwc"
Oct 30 22:30:00.438694 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:00.438568    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/852aabae-2489-4afe-ae9e-20d551cbf9a0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-2b1e88-29889990-mtmwc\" (UID: \"852aabae-2489-4afe-ae9e-20d551cbf9a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2b1e88-29889990-mtmwc"
Oct 30 22:30:00.438769 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:00.438751    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea5c4363-22dd-446e-a6df-1a0ce6497a87-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-3cd77f-29889990-tj449\" (UID: \"ea5c4363-22dd-446e-a6df-1a0ce6497a87\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3cd77f-29889990-tj449"
Oct 30 22:30:00.438844 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:00.438829    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bq2qp\" (UniqueName: \"kubernetes.io/projected/ea5c4363-22dd-446e-a6df-1a0ce6497a87-kube-api-access-bq2qp\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-3cd77f-29889990-tj449\" (UID: \"ea5c4363-22dd-446e-a6df-1a0ce6497a87\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3cd77f-29889990-tj449"
Oct 30 22:30:00.439349 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:00.439326    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/852aabae-2489-4afe-ae9e-20d551cbf9a0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-2b1e88-29889990-mtmwc\" (UID: \"852aabae-2489-4afe-ae9e-20d551cbf9a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2b1e88-29889990-mtmwc"
Oct 30 22:30:00.439446 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:00.439402    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea5c4363-22dd-446e-a6df-1a0ce6497a87-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-3cd77f-29889990-tj449\" (UID: \"ea5c4363-22dd-446e-a6df-1a0ce6497a87\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3cd77f-29889990-tj449"
Oct 30 22:30:00.457203 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:00.457143    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72dtm\" (UniqueName: \"kubernetes.io/projected/852aabae-2489-4afe-ae9e-20d551cbf9a0-kube-api-access-72dtm\") pod \"maas-api-key-cleanup-e2e-worker-w1-2b1e88-29889990-mtmwc\" (UID: \"852aabae-2489-4afe-ae9e-20d551cbf9a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2b1e88-29889990-mtmwc"
Oct 30 22:30:00.457203 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:00.457143    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bq2qp\" (UniqueName: \"kubernetes.io/projected/ea5c4363-22dd-446e-a6df-1a0ce6497a87-kube-api-access-bq2qp\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-3cd77f-29889990-tj449\" (UID: \"ea5c4363-22dd-446e-a6df-1a0ce6497a87\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3cd77f-29889990-tj449"
Oct 30 22:30:00.554217 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:00.554166    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3cd77f-29889990-tj449"
Oct 30 22:30:00.571140 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:00.571108    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2b1e88-29889990-mtmwc"
Oct 30 22:30:00.909504 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:00.909469    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2b1e88-29889990-mtmwc"]
Oct 30 22:30:00.912448 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:30:00.912418    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod852aabae_2489_4afe_ae9e_20d551cbf9a0.slice/crio-330787409df2d72e2ddcfd23c68ffde7b31c5d03a4b3c3db0735cd8ff38692f2 WatchSource:0}: Error finding container 330787409df2d72e2ddcfd23c68ffde7b31c5d03a4b3c3db0735cd8ff38692f2: Status 404 returned error can't find the container with id 330787409df2d72e2ddcfd23c68ffde7b31c5d03a4b3c3db0735cd8ff38692f2
Oct 30 22:30:00.922161 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:00.922137    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3cd77f-29889990-tj449"]
Oct 30 22:30:00.924964 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:30:00.924942    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea5c4363_22dd_446e_a6df_1a0ce6497a87.slice/crio-2ee0382d75a88c19935c0b7dd446c8678beb4994740ca9de22ac11b911622142 WatchSource:0}: Error finding container 2ee0382d75a88c19935c0b7dd446c8678beb4994740ca9de22ac11b911622142: Status 404 returned error can't find the container with id 2ee0382d75a88c19935c0b7dd446c8678beb4994740ca9de22ac11b911622142
Oct 30 22:30:01.075128 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:01.075094    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2b1e88-29889990-mtmwc" event={"ID":"852aabae-2489-4afe-ae9e-20d551cbf9a0","Type":"ContainerStarted","Data":"330787409df2d72e2ddcfd23c68ffde7b31c5d03a4b3c3db0735cd8ff38692f2"}
Oct 30 22:30:01.076783 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:01.076758    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3cd77f-29889990-tj449" event={"ID":"ea5c4363-22dd-446e-a6df-1a0ce6497a87","Type":"ContainerStarted","Data":"2ee0382d75a88c19935c0b7dd446c8678beb4994740ca9de22ac11b911622142"}
Oct 30 22:30:02.085293 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:02.085206    4477 generic.go:356] "Generic (PLEG): container finished" podID="852aabae-2489-4afe-ae9e-20d551cbf9a0" containerID="f20c1fda9efb1e7534bf2100ab3f1c09d3325d33545b8585882f175c7d47e5ab" exitCode=60
Oct 30 22:30:02.085767 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:02.085291    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2b1e88-29889990-mtmwc" event={"ID":"852aabae-2489-4afe-ae9e-20d551cbf9a0","Type":"ContainerDied","Data":"f20c1fda9efb1e7534bf2100ab3f1c09d3325d33545b8585882f175c7d47e5ab"}
Oct 30 22:30:02.085767 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:02.085689    4477 scope.go:122] "RemoveContainer" containerID="f20c1fda9efb1e7534bf2100ab3f1c09d3325d33545b8585882f175c7d47e5ab"
Oct 30 22:30:02.087222 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:02.087199    4477 generic.go:356] "Generic (PLEG): container finished" podID="ea5c4363-22dd-446e-a6df-1a0ce6497a87" containerID="35e18807a3e7184ce44ac886e780be9097e00db8f3217d586b6d134f1e953067" exitCode=60
Oct 30 22:30:02.087325 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:02.087285    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3cd77f-29889990-tj449" event={"ID":"ea5c4363-22dd-446e-a6df-1a0ce6497a87","Type":"ContainerDied","Data":"35e18807a3e7184ce44ac886e780be9097e00db8f3217d586b6d134f1e953067"}
Oct 30 22:30:02.087513 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:02.087494    4477 scope.go:122] "RemoveContainer" containerID="35e18807a3e7184ce44ac886e780be9097e00db8f3217d586b6d134f1e953067"
Oct 30 22:30:03.097385 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:03.097336    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3cd77f-29889990-tj449" event={"ID":"ea5c4363-22dd-446e-a6df-1a0ce6497a87","Type":"ContainerStarted","Data":"017fd0a91699904bcc1b31be71d7085645ea01a9b251c9431cdd5582853ff09b"}
Oct 30 22:30:03.099960 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:03.099914    4477 generic.go:356] "Generic (PLEG): container finished" podID="852aabae-2489-4afe-ae9e-20d551cbf9a0" containerID="f4c2dd3aa1849171c3d861833e06e721d386658280e811fed1c57afab6416c2f" exitCode=60
Oct 30 22:30:03.100181 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:03.099952    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2b1e88-29889990-mtmwc" event={"ID":"852aabae-2489-4afe-ae9e-20d551cbf9a0","Type":"ContainerDied","Data":"f4c2dd3aa1849171c3d861833e06e721d386658280e811fed1c57afab6416c2f"}
Oct 30 22:30:03.100181 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:03.100024    4477 scope.go:122] "RemoveContainer" containerID="f20c1fda9efb1e7534bf2100ab3f1c09d3325d33545b8585882f175c7d47e5ab"
Oct 30 22:30:03.100317 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:03.100298    4477 scope.go:122] "RemoveContainer" containerID="f4c2dd3aa1849171c3d861833e06e721d386658280e811fed1c57afab6416c2f"
Oct 30 22:30:03.100700 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:30:03.100680    4477 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-2b1e88-29889990-mtmwc_odh-ai-gateway-infra(852aabae-2489-4afe-ae9e-20d551cbf9a0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2b1e88-29889990-mtmwc" podUID="852aabae-2489-4afe-ae9e-20d551cbf9a0"
Oct 30 22:30:03.133860 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:03.133650    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3cd77f-29889990-tj449" podStartSLOduration=3.133631503 podStartE2EDuration="3.133631503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:03.131995494 +0000 UTC m=+1387.057921723" watchObservedRunningTime="2026-10-30 22:30:03.133631503 +0000 UTC m=+1387.059557710"
Oct 30 22:30:04.108396 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:04.108369    4477 scope.go:122] "RemoveContainer" containerID="f4c2dd3aa1849171c3d861833e06e721d386658280e811fed1c57afab6416c2f"
Oct 30 22:30:04.108838 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:30:04.108700    4477 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-2b1e88-29889990-mtmwc_odh-ai-gateway-infra(852aabae-2489-4afe-ae9e-20d551cbf9a0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2b1e88-29889990-mtmwc" podUID="852aabae-2489-4afe-ae9e-20d551cbf9a0"
Oct 30 22:30:04.110109 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:04.110083    4477 generic.go:356] "Generic (PLEG): container finished" podID="ea5c4363-22dd-446e-a6df-1a0ce6497a87" containerID="017fd0a91699904bcc1b31be71d7085645ea01a9b251c9431cdd5582853ff09b" exitCode=60
Oct 30 22:30:04.110240 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:04.110207    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3cd77f-29889990-tj449" event={"ID":"ea5c4363-22dd-446e-a6df-1a0ce6497a87","Type":"ContainerDied","Data":"017fd0a91699904bcc1b31be71d7085645ea01a9b251c9431cdd5582853ff09b"}
Oct 30 22:30:04.110287 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:04.110266    4477 scope.go:122] "RemoveContainer" containerID="35e18807a3e7184ce44ac886e780be9097e00db8f3217d586b6d134f1e953067"
Oct 30 22:30:04.110428 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:04.110411    4477 scope.go:122] "RemoveContainer" containerID="017fd0a91699904bcc1b31be71d7085645ea01a9b251c9431cdd5582853ff09b"
Oct 30 22:30:04.110731 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:30:04.110696    4477 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-3cd77f-29889990-tj449_odh-ai-gateway-infra(ea5c4363-22dd-446e-a6df-1a0ce6497a87)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3cd77f-29889990-tj449" podUID="ea5c4363-22dd-446e-a6df-1a0ce6497a87"
Oct 30 22:30:05.120142 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:05.120111    4477 scope.go:122] "RemoveContainer" containerID="017fd0a91699904bcc1b31be71d7085645ea01a9b251c9431cdd5582853ff09b"
Oct 30 22:30:05.120604 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:30:05.120469    4477 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-3cd77f-29889990-tj449_odh-ai-gateway-infra(ea5c4363-22dd-446e-a6df-1a0ce6497a87)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3cd77f-29889990-tj449" podUID="ea5c4363-22dd-446e-a6df-1a0ce6497a87"
Oct 30 22:30:09.128700 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:09.128627    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-gt2jk"]
Oct 30 22:30:09.129149 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:09.128823    4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-gt2jk" podUID="92d165b0-c272-4b3e-969d-ad9dfa6e84d9" containerName="istio-proxy" containerID="cri-o://93d0b2f93136711a14133fa831994d108dd7619ee09468d75a601b44c0a3241d" gracePeriod=30
Oct 30 22:30:09.313580 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:09.313504    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-p4vsn"]
Oct 30 22:30:09.313967 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:09.313835    4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-p4vsn" podUID="7237cb56-76fc-4438-b031-4c0eee080b7d" containerName="istio-proxy" containerID="cri-o://21b0ec2dee5bf78ed90b850ab88070399be834cbb6d18a596fb1972d905f9f81" gracePeriod=30
Oct 30 22:30:09.317683 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:09.317660    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-6wn2g"]
Oct 30 22:30:09.317879 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:09.317853    4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-6wn2g" podUID="720aa32b-9fc1-4013-bd84-a751a9f88fca" containerName="istio-proxy" containerID="cri-o://94cd3a6c54b45c2a429d06dc44bb8f107fb0c0a343256f42c08632476e09b10b" gracePeriod=30
Oct 30 22:30:09.362286 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:09.362254    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-9j5tr"]
Oct 30 22:30:09.362504 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:09.362479    4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-9j5tr" podUID="1fb811da-4dc2-48bb-bec6-afb7a1a80ed2" containerName="istio-proxy" containerID="cri-o://15834bc6d3f73decc1e21123036661a51b2cb8e7864cbf763806851838160302" gracePeriod=30
Oct 30 22:30:09.496832 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:09.496753    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d399fb07-78b4b78864-6x646"]
Oct 30 22:30:09.496978 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:09.496946    4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d399fb07-78b4b78864-6x646" podUID="8f11bbf1-c90c-4e20-a603-93dffec97774" containerName="maas-api" containerID="cri-o://aa382678738ec7916cfa5fb07c528de9a1737c4df0249bd734583c8a7fdd4247" gracePeriod=30
Oct 30 22:30:09.715467 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:09.715411    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-crp67"]
Oct 30 22:30:09.715941 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:09.715890    4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-crp67" podUID="2f5f8741-f496-46a2-b357-72c0e27ea2ae" containerName="istio-proxy" containerID="cri-o://7668f95924f257781ec7ea17f93cba6a28d30ae93598757f6025ae58d55951af" gracePeriod=30
Oct 30 22:30:09.847337 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:09.847302    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d399fb07-78b4b78864-6x646"
Oct 30 22:30:09.885594 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:09.885561    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f11bbf1-c90c-4e20-a603-93dffec97774-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f11bbf1-c90c-4e20-a603-93dffec97774-metrics-certs\") pod \"8f11bbf1-c90c-4e20-a603-93dffec97774\" (UID: \"8f11bbf1-c90c-4e20-a603-93dffec97774\") "
Oct 30 22:30:09.885737 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:09.885641    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f11bbf1-c90c-4e20-a603-93dffec97774-kube-api-access-4css8\" (UniqueName: \"kubernetes.io/projected/8f11bbf1-c90c-4e20-a603-93dffec97774-kube-api-access-4css8\") pod \"8f11bbf1-c90c-4e20-a603-93dffec97774\" (UID: \"8f11bbf1-c90c-4e20-a603-93dffec97774\") "
Oct 30 22:30:09.885737 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:09.885691    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f11bbf1-c90c-4e20-a603-93dffec97774-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f11bbf1-c90c-4e20-a603-93dffec97774-maas-api-tls\") pod \"8f11bbf1-c90c-4e20-a603-93dffec97774\" (UID: \"8f11bbf1-c90c-4e20-a603-93dffec97774\") "
Oct 30 22:30:09.885900 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:09.885877    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f11bbf1-c90c-4e20-a603-93dffec97774-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f11bbf1-c90c-4e20-a603-93dffec97774-openshift-service-ca\") pod \"8f11bbf1-c90c-4e20-a603-93dffec97774\" (UID: \"8f11bbf1-c90c-4e20-a603-93dffec97774\") "
Oct 30 22:30:09.886369 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:09.886342    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f11bbf1-c90c-4e20-a603-93dffec97774-openshift-service-ca" pod "8f11bbf1-c90c-4e20-a603-93dffec97774" (UID: "8f11bbf1-c90c-4e20-a603-93dffec97774"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:09.886652 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:09.886633    4477 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f11bbf1-c90c-4e20-a603-93dffec97774-openshift-service-ca\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:30:09.887976 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:09.887942    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f11bbf1-c90c-4e20-a603-93dffec97774-maas-api-tls" pod "8f11bbf1-c90c-4e20-a603-93dffec97774" (UID: "8f11bbf1-c90c-4e20-a603-93dffec97774"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:30:09.888102 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:09.887979    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f11bbf1-c90c-4e20-a603-93dffec97774-metrics-certs" pod "8f11bbf1-c90c-4e20-a603-93dffec97774" (UID: "8f11bbf1-c90c-4e20-a603-93dffec97774"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:30:09.892576 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:09.892539    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f11bbf1-c90c-4e20-a603-93dffec97774-kube-api-access-4css8" pod "8f11bbf1-c90c-4e20-a603-93dffec97774" (UID: "8f11bbf1-c90c-4e20-a603-93dffec97774"). InnerVolumeSpecName "kube-api-access-4css8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:09.988149 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:09.988109    4477 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f11bbf1-c90c-4e20-a603-93dffec97774-metrics-certs\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:30:09.988149 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:09.988140    4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4css8\" (UniqueName: \"kubernetes.io/projected/8f11bbf1-c90c-4e20-a603-93dffec97774-kube-api-access-4css8\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:30:09.988149 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:09.988153    4477 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f11bbf1-c90c-4e20-a603-93dffec97774-maas-api-tls\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:30:10.182146 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:10.178991    4477 generic.go:356] "Generic (PLEG): container finished" podID="8f11bbf1-c90c-4e20-a603-93dffec97774" containerID="aa382678738ec7916cfa5fb07c528de9a1737c4df0249bd734583c8a7fdd4247" exitCode=0
Oct 30 22:30:10.182146 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:10.179106    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d399fb07-78b4b78864-6x646" event={"ID":"8f11bbf1-c90c-4e20-a603-93dffec97774","Type":"ContainerDied","Data":"aa382678738ec7916cfa5fb07c528de9a1737c4df0249bd734583c8a7fdd4247"}
Oct 30 22:30:10.182146 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:10.179172    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d399fb07-78b4b78864-6x646" event={"ID":"8f11bbf1-c90c-4e20-a603-93dffec97774","Type":"ContainerDied","Data":"f8a46bbb5f1a6b39ad0cdb273fe0ebc1e4612c96e79dfbc97c4441678598e902"}
Oct 30 22:30:10.182146 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:10.179201    4477 scope.go:122] "RemoveContainer" containerID="aa382678738ec7916cfa5fb07c528de9a1737c4df0249bd734583c8a7fdd4247"
Oct 30 22:30:10.182146 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:10.179814    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d399fb07-78b4b78864-6x646"
Oct 30 22:30:10.222371 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:10.222344    4477 scope.go:122] "RemoveContainer" containerID="aa382678738ec7916cfa5fb07c528de9a1737c4df0249bd734583c8a7fdd4247"
Oct 30 22:30:10.222689 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:30:10.222668    4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa382678738ec7916cfa5fb07c528de9a1737c4df0249bd734583c8a7fdd4247\": container with ID starting with aa382678738ec7916cfa5fb07c528de9a1737c4df0249bd734583c8a7fdd4247 not found: ID does not exist" containerID="aa382678738ec7916cfa5fb07c528de9a1737c4df0249bd734583c8a7fdd4247"
Oct 30 22:30:10.222767 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:10.222705    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa382678738ec7916cfa5fb07c528de9a1737c4df0249bd734583c8a7fdd4247"} err="failed to get container status \"aa382678738ec7916cfa5fb07c528de9a1737c4df0249bd734583c8a7fdd4247\": rpc error: code = NotFound desc = could not find container \"aa382678738ec7916cfa5fb07c528de9a1737c4df0249bd734583c8a7fdd4247\": container with ID starting with aa382678738ec7916cfa5fb07c528de9a1737c4df0249bd734583c8a7fdd4247 not found: ID does not exist"
Oct 30 22:30:10.258108 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:10.257689    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d399fb07-78b4b78864-6x646"]
Oct 30 22:30:10.265977 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:10.265948    4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d399fb07-78b4b78864-6x646"]
Oct 30 22:30:12.199388 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:12.199354    4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f11bbf1-c90c-4e20-a603-93dffec97774" path="/var/lib/kubelet/pods/8f11bbf1-c90c-4e20-a603-93dffec97774/volumes"
Oct 30 22:30:14.571375 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:14.571344    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-gt2jk"
Oct 30 22:30:14.652969 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:14.652930    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-p4vsn"
Oct 30 22:30:14.661847 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:14.661820    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92d165b0-c272-4b3e-969d-ad9dfa6e84d9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92d165b0-c272-4b3e-969d-ad9dfa6e84d9-istiod-ca-cert\") pod \"92d165b0-c272-4b3e-969d-ad9dfa6e84d9\" (UID: \"92d165b0-c272-4b3e-969d-ad9dfa6e84d9\") "
Oct 30 22:30:14.662014 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:14.661878    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92d165b0-c272-4b3e-969d-ad9dfa6e84d9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92d165b0-c272-4b3e-969d-ad9dfa6e84d9-istio-envoy\") pod \"92d165b0-c272-4b3e-969d-ad9dfa6e84d9\" (UID: \"92d165b0-c272-4b3e-969d-ad9dfa6e84d9\") "
Oct 30 22:30:14.662014 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:14.661967    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92d165b0-c272-4b3e-969d-ad9dfa6e84d9-kube-api-access-lbwff\" (UniqueName: \"kubernetes.io/projected/92d165b0-c272-4b3e-969d-ad9dfa6e84d9-kube-api-access-lbwff\") pod \"92d165b0-c272-4b3e-969d-ad9dfa6e84d9\" (UID: \"92d165b0-c272-4b3e-969d-ad9dfa6e84d9\") "
Oct 30 22:30:14.662151 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:14.662086    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92d165b0-c272-4b3e-969d-ad9dfa6e84d9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92d165b0-c272-4b3e-969d-ad9dfa6e84d9-credential-socket\") pod \"92d165b0-c272-4b3e-969d-ad9dfa6e84d9\" (UID: \"92d165b0-c272-4b3e-969d-ad9dfa6e84d9\") "
Oct 30 22:30:14.662205 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:14.662184    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92d165b0-c272-4b3e-969d-ad9dfa6e84d9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92d165b0-c272-4b3e-969d-ad9dfa6e84d9-istio-data\") pod \"92d165b0-c272-4b3e-969d-ad9dfa6e84d9\" (UID: \"92d165b0-c272-4b3e-969d-ad9dfa6e84d9\") "
Oct 30 22:30:14.664308 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:14.662267    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/92d165b0-c272-4b3e-969d-ad9dfa6e84d9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92d165b0-c272-4b3e-969d-ad9dfa6e84d9-istio-podinfo\") pod \"92d165b0-c272-4b3e-969d-ad9dfa6e84d9\" (UID: \"92d165b0-c272-4b3e-969d-ad9dfa6e84d9\") "
Oct 30 22:30:14.664308 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:14.662359    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92d165b0-c272-4b3e-969d-ad9dfa6e84d9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92d165b0-c272-4b3e-969d-ad9dfa6e84d9-workload-socket\") pod \"92d165b0-c272-4b3e-969d-ad9dfa6e84d9\" (UID: \"92d165b0-c272-4b3e-969d-ad9dfa6e84d9\") "
Oct 30 22:30:14.664308 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:14.662393    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92d165b0-c272-4b3e-969d-ad9dfa6e84d9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92d165b0-c272-4b3e-969d-ad9dfa6e84d9-workload-certs\") pod \"92d165b0-c272-4b3e-969d-ad9dfa6e84d9\" (UID: \"92d165b0-c272-4b3e-969d-ad9dfa6e84d9\") "
Oct 30 22:30:14.664308 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:14.662441    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92d165b0-c272-4b3e-969d-ad9dfa6e84d9-istio-token\" (UniqueName: \"kubernetes.io/projected/92d165b0-c272-4b3e-969d-ad9dfa6e84d9-istio-token\") pod \"92d165b0-c272-4b3e-969d-ad9dfa6e84d9\" (UID: \"92d165b0-c272-4b3e-969d-ad9dfa6e84d9\") "
Oct 30 22:30:14.664308 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:14.663554    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92d165b0-c272-4b3e-969d-ad9dfa6e84d9-istiod-ca-cert" pod "92d165b0-c272-4b3e-969d-ad9dfa6e84d9" (UID: "92d165b0-c272-4b3e-969d-ad9dfa6e84d9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:14.664308 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:14.663803    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92d165b0-c272-4b3e-969d-ad9dfa6e84d9-istio-data" pod "92d165b0-c272-4b3e-969d-ad9dfa6e84d9" (UID: "92d165b0-c272-4b3e-969d-ad9dfa6e84d9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:14.664308 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:14.664165    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92d165b0-c272-4b3e-969d-ad9dfa6e84d9-workload-socket" pod "92d165b0-c272-4b3e-969d-ad9dfa6e84d9" (UID: "92d165b0-c272-4b3e-969d-ad9dfa6e84d9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:14.664308 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:14.664231    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92d165b0-c272-4b3e-969d-ad9dfa6e84d9-credential-socket" pod "92d165b0-c272-4b3e-969d-ad9dfa6e84d9" (UID: "92d165b0-c272-4b3e-969d-ad9dfa6e84d9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:14.664308 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:14.664276    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92d165b0-c272-4b3e-969d-ad9dfa6e84d9-workload-certs" pod "92d165b0-c272-4b3e-969d-ad9dfa6e84d9" (UID: "92d165b0-c272-4b3e-969d-ad9dfa6e84d9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:14.676413 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:14.670015    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92d165b0-c272-4b3e-969d-ad9dfa6e84d9-istio-envoy" pod "92d165b0-c272-4b3e-969d-ad9dfa6e84d9" (UID: "92d165b0-c272-4b3e-969d-ad9dfa6e84d9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:14.688743 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:14.688680    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92d165b0-c272-4b3e-969d-ad9dfa6e84d9-istio-token" pod "92d165b0-c272-4b3e-969d-ad9dfa6e84d9" (UID: "92d165b0-c272-4b3e-969d-ad9dfa6e84d9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:14.690947 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:14.690909    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92d165b0-c272-4b3e-969d-ad9dfa6e84d9-kube-api-access-lbwff" pod "92d165b0-c272-4b3e-969d-ad9dfa6e84d9" (UID: "92d165b0-c272-4b3e-969d-ad9dfa6e84d9"). InnerVolumeSpecName "kube-api-access-lbwff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:14.692771 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:14.692728    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/92d165b0-c272-4b3e-969d-ad9dfa6e84d9-istio-podinfo" pod "92d165b0-c272-4b3e-969d-ad9dfa6e84d9" (UID: "92d165b0-c272-4b3e-969d-ad9dfa6e84d9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:30:14.720115 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:14.720082    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-6wn2g"
Oct 30 22:30:14.765508 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:14.765475    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7237cb56-76fc-4438-b031-4c0eee080b7d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7237cb56-76fc-4438-b031-4c0eee080b7d-workload-socket\") pod \"7237cb56-76fc-4438-b031-4c0eee080b7d\" (UID: \"7237cb56-76fc-4438-b031-4c0eee080b7d\") "
Oct 30 22:30:14.765702 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:14.765584    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7237cb56-76fc-4438-b031-4c0eee080b7d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7237cb56-76fc-4438-b031-4c0eee080b7d-istiod-ca-cert\") pod \"7237cb56-76fc-4438-b031-4c0eee080b7d\" (UID: \"7237cb56-76fc-4438-b031-4c0eee080b7d\") "
Oct 30 22:30:14.765702 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:14.765650    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7237cb56-76fc-4438-b031-4c0eee080b7d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7237cb56-76fc-4438-b031-4c0eee080b7d-istio-envoy\") pod \"7237cb56-76fc-4438-b031-4c0eee080b7d\" (UID: \"7237cb56-76fc-4438-b031-4c0eee080b7d\") "
Oct 30 22:30:14.765862 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:14.765716    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7237cb56-76fc-4438-b031-4c0eee080b7d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7237cb56-76fc-4438-b031-4c0eee080b7d-workload-certs\") pod \"7237cb56-76fc-4438-b031-4c0eee080b7d\" (UID: \"7237cb56-76fc-4438-b031-4c0eee080b7d\") "
Oct 30 22:30:14.765862 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:14.765829    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7237cb56-76fc-4438-b031-4c0eee080b7d-istio-token\" (UniqueName: \"kubernetes.io/projected/7237cb56-76fc-4438-b031-4c0eee080b7d-istio-token\") pod \"7237cb56-76fc-4438-b031-4c0eee080b7d\" (UID: \"7237cb56-76fc-4438-b031-4c0eee080b7d\") "
Oct 30 22:30:14.766005 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:14.765858    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7237cb56-76fc-4438-b031-4c0eee080b7d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7237cb56-76fc-4438-b031-4c0eee080b7d-istio-data\") pod \"7237cb56-76fc-4438-b031-4c0eee080b7d\" (UID: \"7237cb56-76fc-4438-b031-4c0eee080b7d\") "
Oct 30 22:30:14.766005 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:14.765976    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7237cb56-76fc-4438-b031-4c0eee080b7d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7237cb56-76fc-4438-b031-4c0eee080b7d-credential-socket\") pod \"7237cb56-76fc-4438-b031-4c0eee080b7d\" (UID: \"7237cb56-76fc-4438-b031-4c0eee080b7d\") "
Oct 30 22:30:14.766187 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:14.766073    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7237cb56-76fc-4438-b031-4c0eee080b7d-kube-api-access-25gjl\" (UniqueName: \"kubernetes.io/projected/7237cb56-76fc-4438-b031-4c0eee080b7d-kube-api-access-25gjl\") pod \"7237cb56-76fc-4438-b031-4c0eee080b7d\" (UID: \"7237cb56-76fc-4438-b031-4c0eee080b7d\") "
Oct 30 22:30:14.766187 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:14.766153    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7237cb56-76fc-4438-b031-4c0eee080b7d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7237cb56-76fc-4438-b031-4c0eee080b7d-istio-podinfo\") pod \"7237cb56-76fc-4438-b031-4c0eee080b7d\" (UID: \"7237cb56-76fc-4438-b031-4c0eee080b7d\") "
Oct 30 22:30:14.766827 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:14.766806    4477 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92d165b0-c272-4b3e-969d-ad9dfa6e84d9-credential-socket\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:30:14.766900 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:14.766835    4477 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92d165b0-c272-4b3e-969d-ad9dfa6e84d9-istio-data\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:30:14.766900 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:14.766848    4477 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92d165b0-c272-4b3e-969d-ad9dfa6e84d9-istio-podinfo\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:30:14.766900 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:14.766860    4477 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92d165b0-c272-4b3e-969d-ad9dfa6e84d9-workload-socket\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:30:14.766900 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:14.766875    4477 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92d165b0-c272-4b3e-969d-ad9dfa6e84d9-workload-certs\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:30:14.766900 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:14.766890    4477 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92d165b0-c272-4b3e-969d-ad9dfa6e84d9-istio-token\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:30:14.767309 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:14.766902    4477 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92d165b0-c272-4b3e-969d-ad9dfa6e84d9-istiod-ca-cert\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:30:14.767309 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:14.766953    4477 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92d165b0-c272-4b3e-969d-ad9dfa6e84d9-istio-envoy\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:30:14.767309 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:14.766965    4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbwff\" (UniqueName: \"kubernetes.io/projected/92d165b0-c272-4b3e-969d-ad9dfa6e84d9-kube-api-access-lbwff\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:30:14.767679 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:14.767655    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7237cb56-76fc-4438-b031-4c0eee080b7d-workload-certs" pod "7237cb56-76fc-4438-b031-4c0eee080b7d" (UID: "7237cb56-76fc-4438-b031-4c0eee080b7d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:14.767892 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:14.767858    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7237cb56-76fc-4438-b031-4c0eee080b7d-workload-socket" pod "7237cb56-76fc-4438-b031-4c0eee080b7d" (UID: "7237cb56-76fc-4438-b031-4c0eee080b7d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:14.775919 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:14.768160    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7237cb56-76fc-4438-b031-4c0eee080b7d-istiod-ca-cert" pod "7237cb56-76fc-4438-b031-4c0eee080b7d" (UID: "7237cb56-76fc-4438-b031-4c0eee080b7d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:14.775919 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:14.775552    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7237cb56-76fc-4438-b031-4c0eee080b7d-credential-socket" pod "7237cb56-76fc-4438-b031-4c0eee080b7d" (UID: "7237cb56-76fc-4438-b031-4c0eee080b7d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:14.777227 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:14.777201    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7237cb56-76fc-4438-b031-4c0eee080b7d-istio-data" pod "7237cb56-76fc-4438-b031-4c0eee080b7d" (UID: "7237cb56-76fc-4438-b031-4c0eee080b7d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:14.780731 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:14.780694    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7237cb56-76fc-4438-b031-4c0eee080b7d-istio-podinfo" pod "7237cb56-76fc-4438-b031-4c0eee080b7d" (UID: "7237cb56-76fc-4438-b031-4c0eee080b7d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:30:14.793586 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:14.793560    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7237cb56-76fc-4438-b031-4c0eee080b7d-kube-api-access-25gjl" pod "7237cb56-76fc-4438-b031-4c0eee080b7d" (UID: "7237cb56-76fc-4438-b031-4c0eee080b7d"). InnerVolumeSpecName "kube-api-access-25gjl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:14.794749 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:14.793848    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7237cb56-76fc-4438-b031-4c0eee080b7d-istio-envoy" pod "7237cb56-76fc-4438-b031-4c0eee080b7d" (UID: "7237cb56-76fc-4438-b031-4c0eee080b7d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:14.794749 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:14.793907    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7237cb56-76fc-4438-b031-4c0eee080b7d-istio-token" pod "7237cb56-76fc-4438-b031-4c0eee080b7d" (UID: "7237cb56-76fc-4438-b031-4c0eee080b7d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:14.836476 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:14.836455    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-9j5tr"
Oct 30 22:30:14.868275 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:14.868253    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/720aa32b-9fc1-4013-bd84-a751a9f88fca-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/720aa32b-9fc1-4013-bd84-a751a9f88fca-istio-data\") pod \"720aa32b-9fc1-4013-bd84-a751a9f88fca\" (UID: \"720aa32b-9fc1-4013-bd84-a751a9f88fca\") "
Oct 30 22:30:14.868379 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:14.868339    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/720aa32b-9fc1-4013-bd84-a751a9f88fca-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/720aa32b-9fc1-4013-bd84-a751a9f88fca-workload-socket\") pod \"720aa32b-9fc1-4013-bd84-a751a9f88fca\" (UID: \"720aa32b-9fc1-4013-bd84-a751a9f88fca\") "
Oct 30 22:30:14.868427 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:14.868411    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/720aa32b-9fc1-4013-bd84-a751a9f88fca-kube-api-access-gpqdf\" (UniqueName: \"kubernetes.io/projected/720aa32b-9fc1-4013-bd84-a751a9f88fca-kube-api-access-gpqdf\") pod \"720aa32b-9fc1-4013-bd84-a751a9f88fca\" (UID: \"720aa32b-9fc1-4013-bd84-a751a9f88fca\") "
Oct 30 22:30:14.868464 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:14.868444    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/720aa32b-9fc1-4013-bd84-a751a9f88fca-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/720aa32b-9fc1-4013-bd84-a751a9f88fca-credential-socket\") pod \"720aa32b-9fc1-4013-bd84-a751a9f88fca\" (UID: \"720aa32b-9fc1-4013-bd84-a751a9f88fca\") "
Oct 30 22:30:14.868512 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:14.868490    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/720aa32b-9fc1-4013-bd84-a751a9f88fca-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/720aa32b-9fc1-4013-bd84-a751a9f88fca-workload-certs\") pod \"720aa32b-9fc1-4013-bd84-a751a9f88fca\" (UID: \"720aa32b-9fc1-4013-bd84-a751a9f88fca\") "
Oct 30 22:30:14.868577 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:14.868537    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/720aa32b-9fc1-4013-bd84-a751a9f88fca-istio-token\" (UniqueName: \"kubernetes.io/projected/720aa32b-9fc1-4013-bd84-a751a9f88fca-istio-token\") pod \"720aa32b-9fc1-4013-bd84-a751a9f88fca\" (UID: \"720aa32b-9fc1-4013-bd84-a751a9f88fca\") "
Oct 30 22:30:14.868633 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:14.868548    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/720aa32b-9fc1-4013-bd84-a751a9f88fca-istio-data" pod "720aa32b-9fc1-4013-bd84-a751a9f88fca" (UID: "720aa32b-9fc1-4013-bd84-a751a9f88fca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:14.868688 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:14.868640    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/720aa32b-9fc1-4013-bd84-a751a9f88fca-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/720aa32b-9fc1-4013-bd84-a751a9f88fca-istio-podinfo\") pod \"720aa32b-9fc1-4013-bd84-a751a9f88fca\" (UID: \"720aa32b-9fc1-4013-bd84-a751a9f88fca\") "
Oct 30 22:30:14.868753 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:14.868729    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/720aa32b-9fc1-4013-bd84-a751a9f88fca-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/720aa32b-9fc1-4013-bd84-a751a9f88fca-istio-envoy\") pod \"720aa32b-9fc1-4013-bd84-a751a9f88fca\" (UID: \"720aa32b-9fc1-4013-bd84-a751a9f88fca\") "
Oct 30 22:30:14.868814 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:14.868636    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/720aa32b-9fc1-4013-bd84-a751a9f88fca-workload-socket" pod "720aa32b-9fc1-4013-bd84-a751a9f88fca" (UID: "720aa32b-9fc1-4013-bd84-a751a9f88fca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:14.868814 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:14.868776    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/720aa32b-9fc1-4013-bd84-a751a9f88fca-credential-socket" pod "720aa32b-9fc1-4013-bd84-a751a9f88fca" (UID: "720aa32b-9fc1-4013-bd84-a751a9f88fca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:14.868987 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:14.868814    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/720aa32b-9fc1-4013-bd84-a751a9f88fca-workload-certs" pod "720aa32b-9fc1-4013-bd84-a751a9f88fca" (UID: "720aa32b-9fc1-4013-bd84-a751a9f88fca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:14.868987 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:14.868858    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/720aa32b-9fc1-4013-bd84-a751a9f88fca-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/720aa32b-9fc1-4013-bd84-a751a9f88fca-istiod-ca-cert\") pod \"720aa32b-9fc1-4013-bd84-a751a9f88fca\" (UID: \"720aa32b-9fc1-4013-bd84-a751a9f88fca\") "
Oct 30 22:30:14.869303 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:14.869283    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/720aa32b-9fc1-4013-bd84-a751a9f88fca-istiod-ca-cert" pod "720aa32b-9fc1-4013-bd84-a751a9f88fca" (UID: "720aa32b-9fc1-4013-bd84-a751a9f88fca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:14.869594 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:14.869571    4477 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7237cb56-76fc-4438-b031-4c0eee080b7d-workload-socket\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:30:14.869594 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:14.869590    4477 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7237cb56-76fc-4438-b031-4c0eee080b7d-istiod-ca-cert\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:30:14.869684 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:14.869604    4477 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7237cb56-76fc-4438-b031-4c0eee080b7d-istio-envoy\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:30:14.869684 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:14.869616    4477 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7237cb56-76fc-4438-b031-4c0eee080b7d-workload-certs\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:30:14.869684 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:14.869624    4477 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/720aa32b-9fc1-4013-bd84-a751a9f88fca-istiod-ca-cert\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:30:14.869684 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:14.869632    4477 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7237cb56-76fc-4438-b031-4c0eee080b7d-istio-token\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:30:14.869684 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:14.869639    4477 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7237cb56-76fc-4438-b031-4c0eee080b7d-istio-data\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:30:14.869684 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:14.869646    4477 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/720aa32b-9fc1-4013-bd84-a751a9f88fca-istio-data\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:30:14.869684 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:14.869653    4477 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/720aa32b-9fc1-4013-bd84-a751a9f88fca-workload-socket\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:30:14.869684 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:14.869660    4477 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7237cb56-76fc-4438-b031-4c0eee080b7d-credential-socket\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:30:14.869684 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:14.869670    4477 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/720aa32b-9fc1-4013-bd84-a751a9f88fca-credential-socket\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:30:14.869684 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:14.869684    4477 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/720aa32b-9fc1-4013-bd84-a751a9f88fca-workload-certs\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:30:14.870153 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:14.869693    4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25gjl\" (UniqueName: \"kubernetes.io/projected/7237cb56-76fc-4438-b031-4c0eee080b7d-kube-api-access-25gjl\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:30:14.870153 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:14.869701    4477 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7237cb56-76fc-4438-b031-4c0eee080b7d-istio-podinfo\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:30:14.871079 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:14.871033    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/720aa32b-9fc1-4013-bd84-a751a9f88fca-istio-podinfo" pod "720aa32b-9fc1-4013-bd84-a751a9f88fca" (UID: "720aa32b-9fc1-4013-bd84-a751a9f88fca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:30:14.871177 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:14.871138    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/720aa32b-9fc1-4013-bd84-a751a9f88fca-kube-api-access-gpqdf" pod "720aa32b-9fc1-4013-bd84-a751a9f88fca" (UID: "720aa32b-9fc1-4013-bd84-a751a9f88fca"). InnerVolumeSpecName "kube-api-access-gpqdf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:14.871287 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:14.871266    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/720aa32b-9fc1-4013-bd84-a751a9f88fca-istio-envoy" pod "720aa32b-9fc1-4013-bd84-a751a9f88fca" (UID: "720aa32b-9fc1-4013-bd84-a751a9f88fca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:14.883849 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:14.883707    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/720aa32b-9fc1-4013-bd84-a751a9f88fca-istio-token" pod "720aa32b-9fc1-4013-bd84-a751a9f88fca" (UID: "720aa32b-9fc1-4013-bd84-a751a9f88fca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:14.970940 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:14.970869    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fb811da-4dc2-48bb-bec6-afb7a1a80ed2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fb811da-4dc2-48bb-bec6-afb7a1a80ed2-workload-socket\") pod \"1fb811da-4dc2-48bb-bec6-afb7a1a80ed2\" (UID: \"1fb811da-4dc2-48bb-bec6-afb7a1a80ed2\") "
Oct 30 22:30:14.971122 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:14.971017    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fb811da-4dc2-48bb-bec6-afb7a1a80ed2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fb811da-4dc2-48bb-bec6-afb7a1a80ed2-credential-socket\") pod \"1fb811da-4dc2-48bb-bec6-afb7a1a80ed2\" (UID: \"1fb811da-4dc2-48bb-bec6-afb7a1a80ed2\") "
Oct 30 22:30:14.971191 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:14.971125    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fb811da-4dc2-48bb-bec6-afb7a1a80ed2-kube-api-access-rqqcm\" (UniqueName: \"kubernetes.io/projected/1fb811da-4dc2-48bb-bec6-afb7a1a80ed2-kube-api-access-rqqcm\") pod \"1fb811da-4dc2-48bb-bec6-afb7a1a80ed2\" (UID: \"1fb811da-4dc2-48bb-bec6-afb7a1a80ed2\") "
Oct 30 22:30:14.971191 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:14.971164    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fb811da-4dc2-48bb-bec6-afb7a1a80ed2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fb811da-4dc2-48bb-bec6-afb7a1a80ed2-istiod-ca-cert\") pod \"1fb811da-4dc2-48bb-bec6-afb7a1a80ed2\" (UID: \"1fb811da-4dc2-48bb-bec6-afb7a1a80ed2\") "
Oct 30 22:30:14.971299 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:14.971201    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fb811da-4dc2-48bb-bec6-afb7a1a80ed2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fb811da-4dc2-48bb-bec6-afb7a1a80ed2-istio-data\") pod \"1fb811da-4dc2-48bb-bec6-afb7a1a80ed2\" (UID: \"1fb811da-4dc2-48bb-bec6-afb7a1a80ed2\") "
Oct 30 22:30:14.971353 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:14.971314    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fb811da-4dc2-48bb-bec6-afb7a1a80ed2-istio-token\" (UniqueName: \"kubernetes.io/projected/1fb811da-4dc2-48bb-bec6-afb7a1a80ed2-istio-token\") pod \"1fb811da-4dc2-48bb-bec6-afb7a1a80ed2\" (UID: \"1fb811da-4dc2-48bb-bec6-afb7a1a80ed2\") "
Oct 30 22:30:14.971414 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:14.971343    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fb811da-4dc2-48bb-bec6-afb7a1a80ed2-credential-socket" pod "1fb811da-4dc2-48bb-bec6-afb7a1a80ed2" (UID: "1fb811da-4dc2-48bb-bec6-afb7a1a80ed2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:14.971414 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:14.971356    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1fb811da-4dc2-48bb-bec6-afb7a1a80ed2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fb811da-4dc2-48bb-bec6-afb7a1a80ed2-istio-podinfo\") pod \"1fb811da-4dc2-48bb-bec6-afb7a1a80ed2\" (UID: \"1fb811da-4dc2-48bb-bec6-afb7a1a80ed2\") "
Oct 30 22:30:14.971414 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:14.971379    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fb811da-4dc2-48bb-bec6-afb7a1a80ed2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fb811da-4dc2-48bb-bec6-afb7a1a80ed2-istio-envoy\") pod \"1fb811da-4dc2-48bb-bec6-afb7a1a80ed2\" (UID: \"1fb811da-4dc2-48bb-bec6-afb7a1a80ed2\") "
Oct 30 22:30:14.971571 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:14.971423    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fb811da-4dc2-48bb-bec6-afb7a1a80ed2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fb811da-4dc2-48bb-bec6-afb7a1a80ed2-workload-certs\") pod \"1fb811da-4dc2-48bb-bec6-afb7a1a80ed2\" (UID: \"1fb811da-4dc2-48bb-bec6-afb7a1a80ed2\") "
Oct 30 22:30:14.971571 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:14.971516    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fb811da-4dc2-48bb-bec6-afb7a1a80ed2-workload-socket" pod "1fb811da-4dc2-48bb-bec6-afb7a1a80ed2" (UID: "1fb811da-4dc2-48bb-bec6-afb7a1a80ed2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:14.971711 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:14.971642    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fb811da-4dc2-48bb-bec6-afb7a1a80ed2-istio-data" pod "1fb811da-4dc2-48bb-bec6-afb7a1a80ed2" (UID: "1fb811da-4dc2-48bb-bec6-afb7a1a80ed2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:14.972293 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:14.972265    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fb811da-4dc2-48bb-bec6-afb7a1a80ed2-workload-certs" pod "1fb811da-4dc2-48bb-bec6-afb7a1a80ed2" (UID: "1fb811da-4dc2-48bb-bec6-afb7a1a80ed2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:14.972434 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:14.972376    4477 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/720aa32b-9fc1-4013-bd84-a751a9f88fca-istio-envoy\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:30:14.972434 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:14.972406    4477 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fb811da-4dc2-48bb-bec6-afb7a1a80ed2-credential-socket\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:30:14.972434 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:14.972426    4477 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fb811da-4dc2-48bb-bec6-afb7a1a80ed2-istio-data\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:30:14.972636 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:14.972442    4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpqdf\" (UniqueName: \"kubernetes.io/projected/720aa32b-9fc1-4013-bd84-a751a9f88fca-kube-api-access-gpqdf\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:30:14.972636 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:14.972458    4477 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/720aa32b-9fc1-4013-bd84-a751a9f88fca-istio-token\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:30:14.972636 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:14.972473    4477 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/720aa32b-9fc1-4013-bd84-a751a9f88fca-istio-podinfo\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:30:14.972636 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:14.972488    4477 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fb811da-4dc2-48bb-bec6-afb7a1a80ed2-workload-socket\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:30:14.972636 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:14.972512    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fb811da-4dc2-48bb-bec6-afb7a1a80ed2-istiod-ca-cert" pod "1fb811da-4dc2-48bb-bec6-afb7a1a80ed2" (UID: "1fb811da-4dc2-48bb-bec6-afb7a1a80ed2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:14.974698 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:14.974668    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fb811da-4dc2-48bb-bec6-afb7a1a80ed2-istio-token" pod "1fb811da-4dc2-48bb-bec6-afb7a1a80ed2" (UID: "1fb811da-4dc2-48bb-bec6-afb7a1a80ed2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:14.974698 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:14.974693    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1fb811da-4dc2-48bb-bec6-afb7a1a80ed2-istio-podinfo" pod "1fb811da-4dc2-48bb-bec6-afb7a1a80ed2" (UID: "1fb811da-4dc2-48bb-bec6-afb7a1a80ed2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:30:14.974885 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:14.974672    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fb811da-4dc2-48bb-bec6-afb7a1a80ed2-istio-envoy" pod "1fb811da-4dc2-48bb-bec6-afb7a1a80ed2" (UID: "1fb811da-4dc2-48bb-bec6-afb7a1a80ed2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:14.974885 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:14.974725    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fb811da-4dc2-48bb-bec6-afb7a1a80ed2-kube-api-access-rqqcm" pod "1fb811da-4dc2-48bb-bec6-afb7a1a80ed2" (UID: "1fb811da-4dc2-48bb-bec6-afb7a1a80ed2"). InnerVolumeSpecName "kube-api-access-rqqcm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:15.038620 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:15.038597    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-crp67"
Oct 30 22:30:15.073499 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:15.073473    4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqqcm\" (UniqueName: \"kubernetes.io/projected/1fb811da-4dc2-48bb-bec6-afb7a1a80ed2-kube-api-access-rqqcm\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:30:15.073499 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:15.073504    4477 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fb811da-4dc2-48bb-bec6-afb7a1a80ed2-istiod-ca-cert\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:30:15.073671 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:15.073521    4477 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fb811da-4dc2-48bb-bec6-afb7a1a80ed2-istio-token\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:30:15.073671 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:15.073534    4477 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fb811da-4dc2-48bb-bec6-afb7a1a80ed2-istio-podinfo\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:30:15.073671 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:15.073547    4477 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fb811da-4dc2-48bb-bec6-afb7a1a80ed2-istio-envoy\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:30:15.073671 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:15.073559    4477 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fb811da-4dc2-48bb-bec6-afb7a1a80ed2-workload-certs\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:30:15.182504 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:15.182474    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2f5f8741-f496-46a2-b357-72c0e27ea2ae-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f5f8741-f496-46a2-b357-72c0e27ea2ae-istio-podinfo\") pod \"2f5f8741-f496-46a2-b357-72c0e27ea2ae\" (UID: \"2f5f8741-f496-46a2-b357-72c0e27ea2ae\") "
Oct 30 22:30:15.182676 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:15.182562    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f5f8741-f496-46a2-b357-72c0e27ea2ae-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f5f8741-f496-46a2-b357-72c0e27ea2ae-workload-socket\") pod \"2f5f8741-f496-46a2-b357-72c0e27ea2ae\" (UID: \"2f5f8741-f496-46a2-b357-72c0e27ea2ae\") "
Oct 30 22:30:15.182676 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:15.182669    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f5f8741-f496-46a2-b357-72c0e27ea2ae-istio-token\" (UniqueName: \"kubernetes.io/projected/2f5f8741-f496-46a2-b357-72c0e27ea2ae-istio-token\") pod \"2f5f8741-f496-46a2-b357-72c0e27ea2ae\" (UID: \"2f5f8741-f496-46a2-b357-72c0e27ea2ae\") "
Oct 30 22:30:15.182860 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:15.182741    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f5f8741-f496-46a2-b357-72c0e27ea2ae-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f5f8741-f496-46a2-b357-72c0e27ea2ae-istio-envoy\") pod \"2f5f8741-f496-46a2-b357-72c0e27ea2ae\" (UID: \"2f5f8741-f496-46a2-b357-72c0e27ea2ae\") "
Oct 30 22:30:15.182860 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:15.182841    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f5f8741-f496-46a2-b357-72c0e27ea2ae-kube-api-access-k7chh\" (UniqueName: \"kubernetes.io/projected/2f5f8741-f496-46a2-b357-72c0e27ea2ae-kube-api-access-k7chh\") pod \"2f5f8741-f496-46a2-b357-72c0e27ea2ae\" (UID: \"2f5f8741-f496-46a2-b357-72c0e27ea2ae\") "
Oct 30 22:30:15.182977 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:15.182877    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f5f8741-f496-46a2-b357-72c0e27ea2ae-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f5f8741-f496-46a2-b357-72c0e27ea2ae-istio-data\") pod \"2f5f8741-f496-46a2-b357-72c0e27ea2ae\" (UID: \"2f5f8741-f496-46a2-b357-72c0e27ea2ae\") "
Oct 30 22:30:15.183033 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:15.182978    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f5f8741-f496-46a2-b357-72c0e27ea2ae-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f5f8741-f496-46a2-b357-72c0e27ea2ae-istiod-ca-cert\") pod \"2f5f8741-f496-46a2-b357-72c0e27ea2ae\" (UID: \"2f5f8741-f496-46a2-b357-72c0e27ea2ae\") "
Oct 30 22:30:15.183033 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:15.183013    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f5f8741-f496-46a2-b357-72c0e27ea2ae-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f5f8741-f496-46a2-b357-72c0e27ea2ae-workload-certs\") pod \"2f5f8741-f496-46a2-b357-72c0e27ea2ae\" (UID: \"2f5f8741-f496-46a2-b357-72c0e27ea2ae\") "
Oct 30 22:30:15.183256 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:15.183146    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f5f8741-f496-46a2-b357-72c0e27ea2ae-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f5f8741-f496-46a2-b357-72c0e27ea2ae-credential-socket\") pod \"2f5f8741-f496-46a2-b357-72c0e27ea2ae\" (UID: \"2f5f8741-f496-46a2-b357-72c0e27ea2ae\") "
Oct 30 22:30:15.183315 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:15.183301    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f5f8741-f496-46a2-b357-72c0e27ea2ae-workload-socket" pod "2f5f8741-f496-46a2-b357-72c0e27ea2ae" (UID: "2f5f8741-f496-46a2-b357-72c0e27ea2ae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:15.183731 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:15.183689    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f5f8741-f496-46a2-b357-72c0e27ea2ae-istio-data" pod "2f5f8741-f496-46a2-b357-72c0e27ea2ae" (UID: "2f5f8741-f496-46a2-b357-72c0e27ea2ae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:15.183898 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:15.183851    4477 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f5f8741-f496-46a2-b357-72c0e27ea2ae-workload-socket\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:30:15.183960 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:15.183905    4477 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f5f8741-f496-46a2-b357-72c0e27ea2ae-istio-data\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:30:15.184405 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:15.184232    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f5f8741-f496-46a2-b357-72c0e27ea2ae-workload-certs" pod "2f5f8741-f496-46a2-b357-72c0e27ea2ae" (UID: "2f5f8741-f496-46a2-b357-72c0e27ea2ae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:15.184746 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:15.184490    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f5f8741-f496-46a2-b357-72c0e27ea2ae-istiod-ca-cert" pod "2f5f8741-f496-46a2-b357-72c0e27ea2ae" (UID: "2f5f8741-f496-46a2-b357-72c0e27ea2ae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:15.185253 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:15.185182    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f5f8741-f496-46a2-b357-72c0e27ea2ae-credential-socket" pod "2f5f8741-f496-46a2-b357-72c0e27ea2ae" (UID: "2f5f8741-f496-46a2-b357-72c0e27ea2ae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:15.186382 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:15.186357    4477 scope.go:122] "RemoveContainer" containerID="017fd0a91699904bcc1b31be71d7085645ea01a9b251c9431cdd5582853ff09b"
Oct 30 22:30:15.187115 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:15.187075    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f5f8741-f496-46a2-b357-72c0e27ea2ae-istio-token" pod "2f5f8741-f496-46a2-b357-72c0e27ea2ae" (UID: "2f5f8741-f496-46a2-b357-72c0e27ea2ae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:15.195228 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:15.195202    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2f5f8741-f496-46a2-b357-72c0e27ea2ae-istio-podinfo" pod "2f5f8741-f496-46a2-b357-72c0e27ea2ae" (UID: "2f5f8741-f496-46a2-b357-72c0e27ea2ae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:30:15.195328 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:15.195261    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f5f8741-f496-46a2-b357-72c0e27ea2ae-istio-envoy" pod "2f5f8741-f496-46a2-b357-72c0e27ea2ae" (UID: "2f5f8741-f496-46a2-b357-72c0e27ea2ae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:15.196181 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:15.196161    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f5f8741-f496-46a2-b357-72c0e27ea2ae-kube-api-access-k7chh" pod "2f5f8741-f496-46a2-b357-72c0e27ea2ae" (UID: "2f5f8741-f496-46a2-b357-72c0e27ea2ae"). InnerVolumeSpecName "kube-api-access-k7chh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:15.219665 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:15.219637    4477 generic.go:356] "Generic (PLEG): container finished" podID="2f5f8741-f496-46a2-b357-72c0e27ea2ae" containerID="7668f95924f257781ec7ea17f93cba6a28d30ae93598757f6025ae58d55951af" exitCode=0
Oct 30 22:30:15.219769 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:15.219707    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-crp67"
Oct 30 22:30:15.219897 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:15.219866    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-crp67" event={"ID":"2f5f8741-f496-46a2-b357-72c0e27ea2ae","Type":"ContainerDied","Data":"7668f95924f257781ec7ea17f93cba6a28d30ae93598757f6025ae58d55951af"}
Oct 30 22:30:15.220006 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:15.219921    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-crp67" event={"ID":"2f5f8741-f496-46a2-b357-72c0e27ea2ae","Type":"ContainerDied","Data":"c4a23843c1f8e846b7eb82a155d52d5f50bd71c0e69d257d0ec24e2c9d1f16b7"}
Oct 30 22:30:15.220006 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:15.219950    4477 scope.go:122] "RemoveContainer" containerID="7668f95924f257781ec7ea17f93cba6a28d30ae93598757f6025ae58d55951af"
Oct 30 22:30:15.221863 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:15.221706    4477 generic.go:356] "Generic (PLEG): container finished" podID="7237cb56-76fc-4438-b031-4c0eee080b7d" containerID="21b0ec2dee5bf78ed90b850ab88070399be834cbb6d18a596fb1972d905f9f81" exitCode=0
Oct 30 22:30:15.223091 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:15.221999    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-p4vsn"
Oct 30 22:30:15.223091 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:15.222138    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-p4vsn" event={"ID":"7237cb56-76fc-4438-b031-4c0eee080b7d","Type":"ContainerDied","Data":"21b0ec2dee5bf78ed90b850ab88070399be834cbb6d18a596fb1972d905f9f81"}
Oct 30 22:30:15.223091 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:15.222188    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-p4vsn" event={"ID":"7237cb56-76fc-4438-b031-4c0eee080b7d","Type":"ContainerDied","Data":"4a5781e719430479588192547253179cc56118f31525b62c048d926941dc288c"}
Oct 30 22:30:15.224499 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:15.224474    4477 generic.go:356] "Generic (PLEG): container finished" podID="720aa32b-9fc1-4013-bd84-a751a9f88fca" containerID="94cd3a6c54b45c2a429d06dc44bb8f107fb0c0a343256f42c08632476e09b10b" exitCode=0
Oct 30 22:30:15.224607 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:15.224532    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-6wn2g"
Oct 30 22:30:15.224607 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:15.224592    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-6wn2g" event={"ID":"720aa32b-9fc1-4013-bd84-a751a9f88fca","Type":"ContainerDied","Data":"94cd3a6c54b45c2a429d06dc44bb8f107fb0c0a343256f42c08632476e09b10b"}
Oct 30 22:30:15.224710 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:15.224629    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-6wn2g" event={"ID":"720aa32b-9fc1-4013-bd84-a751a9f88fca","Type":"ContainerDied","Data":"942c3e06f8c22f4707ea56c1e3e2573f0ef5a0721fb3bb296e7d8b108139b489"}
Oct 30 22:30:15.225782 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:15.225762    4477 generic.go:356] "Generic (PLEG): container finished" podID="1fb811da-4dc2-48bb-bec6-afb7a1a80ed2" containerID="15834bc6d3f73decc1e21123036661a51b2cb8e7864cbf763806851838160302" exitCode=0
Oct 30 22:30:15.225886 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:15.225856    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-9j5tr" event={"ID":"1fb811da-4dc2-48bb-bec6-afb7a1a80ed2","Type":"ContainerDied","Data":"15834bc6d3f73decc1e21123036661a51b2cb8e7864cbf763806851838160302"}
Oct 30 22:30:15.225953 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:15.225895    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-9j5tr" event={"ID":"1fb811da-4dc2-48bb-bec6-afb7a1a80ed2","Type":"ContainerDied","Data":"d0da360d2cfb4d3c159d34274abb8f6d8fc4d2106af23a8fd9395d15a00462a8"}
Oct 30 22:30:15.226014 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:15.225960    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-9j5tr"
Oct 30 22:30:15.231010 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:15.230990    4477 generic.go:356] "Generic (PLEG): container finished" podID="92d165b0-c272-4b3e-969d-ad9dfa6e84d9" containerID="93d0b2f93136711a14133fa831994d108dd7619ee09468d75a601b44c0a3241d" exitCode=0
Oct 30 22:30:15.231153 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:15.231055    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-gt2jk" event={"ID":"92d165b0-c272-4b3e-969d-ad9dfa6e84d9","Type":"ContainerDied","Data":"93d0b2f93136711a14133fa831994d108dd7619ee09468d75a601b44c0a3241d"}
Oct 30 22:30:15.231153 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:15.231092    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-gt2jk" event={"ID":"92d165b0-c272-4b3e-969d-ad9dfa6e84d9","Type":"ContainerDied","Data":"1afb1e0c09e8a23f2a4401882c31ba0b696eec4befce28db4984bf9a0c9d6330"}
Oct 30 22:30:15.231153 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:15.231144    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-gt2jk"
Oct 30 22:30:15.233835 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:15.233815    4477 scope.go:122] "RemoveContainer" containerID="7668f95924f257781ec7ea17f93cba6a28d30ae93598757f6025ae58d55951af"
Oct 30 22:30:15.234150 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:30:15.234127    4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7668f95924f257781ec7ea17f93cba6a28d30ae93598757f6025ae58d55951af\": container with ID starting with 7668f95924f257781ec7ea17f93cba6a28d30ae93598757f6025ae58d55951af not found: ID does not exist" containerID="7668f95924f257781ec7ea17f93cba6a28d30ae93598757f6025ae58d55951af"
Oct 30 22:30:15.234226 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:15.234158    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7668f95924f257781ec7ea17f93cba6a28d30ae93598757f6025ae58d55951af"} err="failed to get container status \"7668f95924f257781ec7ea17f93cba6a28d30ae93598757f6025ae58d55951af\": rpc error: code = NotFound desc = could not find container \"7668f95924f257781ec7ea17f93cba6a28d30ae93598757f6025ae58d55951af\": container with ID starting with 7668f95924f257781ec7ea17f93cba6a28d30ae93598757f6025ae58d55951af not found: ID does not exist"
Oct 30 22:30:15.234226 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:15.234177    4477 scope.go:122] "RemoveContainer" containerID="21b0ec2dee5bf78ed90b850ab88070399be834cbb6d18a596fb1972d905f9f81"
Oct 30 22:30:15.246963 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:15.246944    4477 scope.go:122] "RemoveContainer" containerID="21b0ec2dee5bf78ed90b850ab88070399be834cbb6d18a596fb1972d905f9f81"
Oct 30 22:30:15.247318 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:30:15.247293    4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21b0ec2dee5bf78ed90b850ab88070399be834cbb6d18a596fb1972d905f9f81\": container with ID starting with 21b0ec2dee5bf78ed90b850ab88070399be834cbb6d18a596fb1972d905f9f81 not found: ID does not exist" containerID="21b0ec2dee5bf78ed90b850ab88070399be834cbb6d18a596fb1972d905f9f81"
Oct 30 22:30:15.247470 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:15.247432    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21b0ec2dee5bf78ed90b850ab88070399be834cbb6d18a596fb1972d905f9f81"} err="failed to get container status \"21b0ec2dee5bf78ed90b850ab88070399be834cbb6d18a596fb1972d905f9f81\": rpc error: code = NotFound desc = could not find container \"21b0ec2dee5bf78ed90b850ab88070399be834cbb6d18a596fb1972d905f9f81\": container with ID starting with 21b0ec2dee5bf78ed90b850ab88070399be834cbb6d18a596fb1972d905f9f81 not found: ID does not exist"
Oct 30 22:30:15.247470 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:15.247464    4477 scope.go:122] "RemoveContainer" containerID="94cd3a6c54b45c2a429d06dc44bb8f107fb0c0a343256f42c08632476e09b10b"
Oct 30 22:30:15.256735 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:15.256542    4477 scope.go:122] "RemoveContainer" containerID="94cd3a6c54b45c2a429d06dc44bb8f107fb0c0a343256f42c08632476e09b10b"
Oct 30 22:30:15.256850 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:30:15.256831    4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94cd3a6c54b45c2a429d06dc44bb8f107fb0c0a343256f42c08632476e09b10b\": container with ID starting with 94cd3a6c54b45c2a429d06dc44bb8f107fb0c0a343256f42c08632476e09b10b not found: ID does not exist" containerID="94cd3a6c54b45c2a429d06dc44bb8f107fb0c0a343256f42c08632476e09b10b"
Oct 30 22:30:15.256896 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:15.256857    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94cd3a6c54b45c2a429d06dc44bb8f107fb0c0a343256f42c08632476e09b10b"} err="failed to get container status \"94cd3a6c54b45c2a429d06dc44bb8f107fb0c0a343256f42c08632476e09b10b\": rpc error: code = NotFound desc = could not find container \"94cd3a6c54b45c2a429d06dc44bb8f107fb0c0a343256f42c08632476e09b10b\": container with ID starting with 94cd3a6c54b45c2a429d06dc44bb8f107fb0c0a343256f42c08632476e09b10b not found: ID does not exist"
Oct 30 22:30:15.256896 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:15.256873    4477 scope.go:122] "RemoveContainer" containerID="15834bc6d3f73decc1e21123036661a51b2cb8e7864cbf763806851838160302"
Oct 30 22:30:15.265922 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:15.265902    4477 scope.go:122] "RemoveContainer" containerID="15834bc6d3f73decc1e21123036661a51b2cb8e7864cbf763806851838160302"
Oct 30 22:30:15.266253 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:30:15.266231    4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15834bc6d3f73decc1e21123036661a51b2cb8e7864cbf763806851838160302\": container with ID starting with 15834bc6d3f73decc1e21123036661a51b2cb8e7864cbf763806851838160302 not found: ID does not exist" containerID="15834bc6d3f73decc1e21123036661a51b2cb8e7864cbf763806851838160302"
Oct 30 22:30:15.266338 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:15.266257    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15834bc6d3f73decc1e21123036661a51b2cb8e7864cbf763806851838160302"} err="failed to get container status \"15834bc6d3f73decc1e21123036661a51b2cb8e7864cbf763806851838160302\": rpc error: code = NotFound desc = could not find container \"15834bc6d3f73decc1e21123036661a51b2cb8e7864cbf763806851838160302\": container with ID starting with 15834bc6d3f73decc1e21123036661a51b2cb8e7864cbf763806851838160302 not found: ID does not exist"
Oct 30 22:30:15.266338 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:15.266271    4477 scope.go:122] "RemoveContainer" containerID="93d0b2f93136711a14133fa831994d108dd7619ee09468d75a601b44c0a3241d"
Oct 30 22:30:15.275390 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:15.275371    4477 scope.go:122] "RemoveContainer" containerID="93d0b2f93136711a14133fa831994d108dd7619ee09468d75a601b44c0a3241d"
Oct 30 22:30:15.275701 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:30:15.275677    4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93d0b2f93136711a14133fa831994d108dd7619ee09468d75a601b44c0a3241d\": container with ID starting with 93d0b2f93136711a14133fa831994d108dd7619ee09468d75a601b44c0a3241d not found: ID does not exist" containerID="93d0b2f93136711a14133fa831994d108dd7619ee09468d75a601b44c0a3241d"
Oct 30 22:30:15.275774 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:15.275705    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93d0b2f93136711a14133fa831994d108dd7619ee09468d75a601b44c0a3241d"} err="failed to get container status \"93d0b2f93136711a14133fa831994d108dd7619ee09468d75a601b44c0a3241d\": rpc error: code = NotFound desc = could not find container \"93d0b2f93136711a14133fa831994d108dd7619ee09468d75a601b44c0a3241d\": container with ID starting with 93d0b2f93136711a14133fa831994d108dd7619ee09468d75a601b44c0a3241d not found: ID does not exist"
Oct 30 22:30:15.286085 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:15.286008    4477 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f5f8741-f496-46a2-b357-72c0e27ea2ae-istio-token\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:30:15.286085 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:15.286083    4477 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f5f8741-f496-46a2-b357-72c0e27ea2ae-istio-envoy\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:30:15.286287 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:15.286100    4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7chh\" (UniqueName: \"kubernetes.io/projected/2f5f8741-f496-46a2-b357-72c0e27ea2ae-kube-api-access-k7chh\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:30:15.286287 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:15.286112    4477 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f5f8741-f496-46a2-b357-72c0e27ea2ae-istiod-ca-cert\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:30:15.286287 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:15.286128    4477 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f5f8741-f496-46a2-b357-72c0e27ea2ae-workload-certs\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:30:15.286287 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:15.286139    4477 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f5f8741-f496-46a2-b357-72c0e27ea2ae-credential-socket\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:30:15.286287 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:15.286154    4477 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f5f8741-f496-46a2-b357-72c0e27ea2ae-istio-podinfo\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:30:15.334188 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:15.334160    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-crp67"]
Oct 30 22:30:15.336845 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:15.336821    4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-crp67"]
Oct 30 22:30:15.366801 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:15.366775    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-p4vsn"]
Oct 30 22:30:15.371109 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:15.371087    4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-p4vsn"]
Oct 30 22:30:15.402875 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:15.402853    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-6wn2g"]
Oct 30 22:30:15.407213 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:15.407190    4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-6wn2g"]
Oct 30 22:30:15.443677 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:15.443657    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-gt2jk"]
Oct 30 22:30:15.450787 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:15.450764    4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-gt2jk"]
Oct 30 22:30:15.485511 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:15.485458    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-9j5tr"]
Oct 30 22:30:15.490322 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:15.490300    4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-9j5tr"]
Oct 30 22:30:16.195210 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:16.195182    4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fb811da-4dc2-48bb-bec6-afb7a1a80ed2" path="/var/lib/kubelet/pods/1fb811da-4dc2-48bb-bec6-afb7a1a80ed2/volumes"
Oct 30 22:30:16.195663 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:16.195649    4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f5f8741-f496-46a2-b357-72c0e27ea2ae" path="/var/lib/kubelet/pods/2f5f8741-f496-46a2-b357-72c0e27ea2ae/volumes"
Oct 30 22:30:16.196123 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:16.196109    4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="720aa32b-9fc1-4013-bd84-a751a9f88fca" path="/var/lib/kubelet/pods/720aa32b-9fc1-4013-bd84-a751a9f88fca/volumes"
Oct 30 22:30:16.196858 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:16.196843    4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7237cb56-76fc-4438-b031-4c0eee080b7d" path="/var/lib/kubelet/pods/7237cb56-76fc-4438-b031-4c0eee080b7d/volumes"
Oct 30 22:30:16.197340 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:16.197325    4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92d165b0-c272-4b3e-969d-ad9dfa6e84d9" path="/var/lib/kubelet/pods/92d165b0-c272-4b3e-969d-ad9dfa6e84d9/volumes"
Oct 30 22:30:16.243423 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:16.243402    4477 generic.go:356] "Generic (PLEG): container finished" podID="ea5c4363-22dd-446e-a6df-1a0ce6497a87" containerID="b28f0b9c9f1689b3cfd9dae7c4d9a9c678909da7c0a38213db19834e6fe1bdb9" exitCode=60
Oct 30 22:30:16.243541 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:16.243467    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3cd77f-29889990-tj449" event={"ID":"ea5c4363-22dd-446e-a6df-1a0ce6497a87","Type":"ContainerDied","Data":"b28f0b9c9f1689b3cfd9dae7c4d9a9c678909da7c0a38213db19834e6fe1bdb9"}
Oct 30 22:30:16.243541 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:16.243511    4477 scope.go:122] "RemoveContainer" containerID="017fd0a91699904bcc1b31be71d7085645ea01a9b251c9431cdd5582853ff09b"
Oct 30 22:30:16.243864 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:16.243841    4477 scope.go:122] "RemoveContainer" containerID="b28f0b9c9f1689b3cfd9dae7c4d9a9c678909da7c0a38213db19834e6fe1bdb9"
Oct 30 22:30:16.244172 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:30:16.244146    4477 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-3cd77f-29889990-tj449_odh-ai-gateway-infra(ea5c4363-22dd-446e-a6df-1a0ce6497a87)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3cd77f-29889990-tj449" podUID="ea5c4363-22dd-446e-a6df-1a0ce6497a87"
Oct 30 22:30:17.186741 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:17.186711    4477 scope.go:122] "RemoveContainer" containerID="f4c2dd3aa1849171c3d861833e06e721d386658280e811fed1c57afab6416c2f"
Oct 30 22:30:17.294014 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:17.293989    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3cd77f-29889990-tj449"]
Oct 30 22:30:17.468098 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:17.468068    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3cd77f-29889990-tj449"
Oct 30 22:30:17.615230 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:17.615199    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea5c4363-22dd-446e-a6df-1a0ce6497a87-kube-api-access-bq2qp\" (UniqueName: \"kubernetes.io/projected/ea5c4363-22dd-446e-a6df-1a0ce6497a87-kube-api-access-bq2qp\") pod \"ea5c4363-22dd-446e-a6df-1a0ce6497a87\" (UID: \"ea5c4363-22dd-446e-a6df-1a0ce6497a87\") "
Oct 30 22:30:17.615393 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:17.615321    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea5c4363-22dd-446e-a6df-1a0ce6497a87-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea5c4363-22dd-446e-a6df-1a0ce6497a87-maas-api-service-ca\") pod \"ea5c4363-22dd-446e-a6df-1a0ce6497a87\" (UID: \"ea5c4363-22dd-446e-a6df-1a0ce6497a87\") "
Oct 30 22:30:17.615671 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:17.615648    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea5c4363-22dd-446e-a6df-1a0ce6497a87-maas-api-service-ca" pod "ea5c4363-22dd-446e-a6df-1a0ce6497a87" (UID: "ea5c4363-22dd-446e-a6df-1a0ce6497a87"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:17.615830 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:17.615813    4477 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea5c4363-22dd-446e-a6df-1a0ce6497a87-maas-api-service-ca\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:30:17.617489 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:17.617464    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea5c4363-22dd-446e-a6df-1a0ce6497a87-kube-api-access-bq2qp" pod "ea5c4363-22dd-446e-a6df-1a0ce6497a87" (UID: "ea5c4363-22dd-446e-a6df-1a0ce6497a87"). InnerVolumeSpecName "kube-api-access-bq2qp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:17.716959 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:17.716883    4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bq2qp\" (UniqueName: \"kubernetes.io/projected/ea5c4363-22dd-446e-a6df-1a0ce6497a87-kube-api-access-bq2qp\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:30:18.261080 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:18.260967    4477 generic.go:356] "Generic (PLEG): container finished" podID="852aabae-2489-4afe-ae9e-20d551cbf9a0" containerID="6e165eb967a52a34e33c12ea0022904410d1f6b0f9a8ef307e4dc5629457889b" exitCode=60
Oct 30 22:30:18.261080 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:18.261052    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2b1e88-29889990-mtmwc" event={"ID":"852aabae-2489-4afe-ae9e-20d551cbf9a0","Type":"ContainerDied","Data":"6e165eb967a52a34e33c12ea0022904410d1f6b0f9a8ef307e4dc5629457889b"}
Oct 30 22:30:18.261301 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:18.261114    4477 scope.go:122] "RemoveContainer" containerID="f4c2dd3aa1849171c3d861833e06e721d386658280e811fed1c57afab6416c2f"
Oct 30 22:30:18.261568 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:18.261538    4477 scope.go:122] "RemoveContainer" containerID="6e165eb967a52a34e33c12ea0022904410d1f6b0f9a8ef307e4dc5629457889b"
Oct 30 22:30:18.261922 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:30:18.261897    4477 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-2b1e88-29889990-mtmwc_odh-ai-gateway-infra(852aabae-2489-4afe-ae9e-20d551cbf9a0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2b1e88-29889990-mtmwc" podUID="852aabae-2489-4afe-ae9e-20d551cbf9a0"
Oct 30 22:30:18.262715 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:18.262696    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3cd77f-29889990-tj449"
Oct 30 22:30:18.262715 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:18.262700    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3cd77f-29889990-tj449" event={"ID":"ea5c4363-22dd-446e-a6df-1a0ce6497a87","Type":"ContainerDied","Data":"2ee0382d75a88c19935c0b7dd446c8678beb4994740ca9de22ac11b911622142"}
Oct 30 22:30:18.290573 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:18.290537    4477 scope.go:122] "RemoveContainer" containerID="b28f0b9c9f1689b3cfd9dae7c4d9a9c678909da7c0a38213db19834e6fe1bdb9"
Oct 30 22:30:18.340609 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:18.340588    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3cd77f-29889990-tj449"]
Oct 30 22:30:18.344276 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:18.344255    4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-3cd77f-29889990-tj449"]
Oct 30 22:30:18.632477 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:18.632445    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-47qsn"]
Oct 30 22:30:18.633547 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:18.633159    4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-47qsn" podUID="a620edfb-2b72-4c1f-8b51-9a13d2cfb485" containerName="istio-proxy" containerID="cri-o://601e77bb3bfbdfa9e4c4e7edd54bac35079d6bccbd64b290c5aee78520417df3" gracePeriod=30
Oct 30 22:30:19.311270 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:19.311233    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2b1e88-29889990-mtmwc"]
Oct 30 22:30:19.504512 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:19.504487    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2b1e88-29889990-mtmwc"
Oct 30 22:30:19.655248 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:19.655224    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/852aabae-2489-4afe-ae9e-20d551cbf9a0-kube-api-access-72dtm\" (UniqueName: \"kubernetes.io/projected/852aabae-2489-4afe-ae9e-20d551cbf9a0-kube-api-access-72dtm\") pod \"852aabae-2489-4afe-ae9e-20d551cbf9a0\" (UID: \"852aabae-2489-4afe-ae9e-20d551cbf9a0\") "
Oct 30 22:30:19.655410 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:19.655372    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/852aabae-2489-4afe-ae9e-20d551cbf9a0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/852aabae-2489-4afe-ae9e-20d551cbf9a0-maas-api-service-ca\") pod \"852aabae-2489-4afe-ae9e-20d551cbf9a0\" (UID: \"852aabae-2489-4afe-ae9e-20d551cbf9a0\") "
Oct 30 22:30:19.655767 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:19.655740    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/852aabae-2489-4afe-ae9e-20d551cbf9a0-maas-api-service-ca" pod "852aabae-2489-4afe-ae9e-20d551cbf9a0" (UID: "852aabae-2489-4afe-ae9e-20d551cbf9a0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:19.656154 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:19.656133    4477 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/852aabae-2489-4afe-ae9e-20d551cbf9a0-maas-api-service-ca\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:30:19.657348 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:19.657323    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/852aabae-2489-4afe-ae9e-20d551cbf9a0-kube-api-access-72dtm" pod "852aabae-2489-4afe-ae9e-20d551cbf9a0" (UID: "852aabae-2489-4afe-ae9e-20d551cbf9a0"). InnerVolumeSpecName "kube-api-access-72dtm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:19.757831 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:19.757794    4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72dtm\" (UniqueName: \"kubernetes.io/projected/852aabae-2489-4afe-ae9e-20d551cbf9a0-kube-api-access-72dtm\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:30:20.210608 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:20.210561    4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea5c4363-22dd-446e-a6df-1a0ce6497a87" path="/var/lib/kubelet/pods/ea5c4363-22dd-446e-a6df-1a0ce6497a87/volumes"
Oct 30 22:30:20.294263 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:20.292506    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2b1e88-29889990-mtmwc" event={"ID":"852aabae-2489-4afe-ae9e-20d551cbf9a0","Type":"ContainerDied","Data":"330787409df2d72e2ddcfd23c68ffde7b31c5d03a4b3c3db0735cd8ff38692f2"}
Oct 30 22:30:20.294263 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:20.292594    4477 scope.go:122] "RemoveContainer" containerID="6e165eb967a52a34e33c12ea0022904410d1f6b0f9a8ef307e4dc5629457889b"
Oct 30 22:30:20.294263 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:20.292738    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2b1e88-29889990-mtmwc"
Oct 30 22:30:20.345720 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:20.345686    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2b1e88-29889990-mtmwc"]
Oct 30 22:30:20.357148 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:20.357116    4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-2b1e88-29889990-mtmwc"]
Oct 30 22:30:22.195765 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:22.195734    4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="852aabae-2489-4afe-ae9e-20d551cbf9a0" path="/var/lib/kubelet/pods/852aabae-2489-4afe-ae9e-20d551cbf9a0/volumes"
Oct 30 22:30:23.921132 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:23.921080    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-47qsn"
Oct 30 22:30:24.022576 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:24.022550    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a620edfb-2b72-4c1f-8b51-9a13d2cfb485-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a620edfb-2b72-4c1f-8b51-9a13d2cfb485-workload-certs\") pod \"a620edfb-2b72-4c1f-8b51-9a13d2cfb485\" (UID: \"a620edfb-2b72-4c1f-8b51-9a13d2cfb485\") "
Oct 30 22:30:24.022766 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:24.022610    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a620edfb-2b72-4c1f-8b51-9a13d2cfb485-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a620edfb-2b72-4c1f-8b51-9a13d2cfb485-istio-envoy\") pod \"a620edfb-2b72-4c1f-8b51-9a13d2cfb485\" (UID: \"a620edfb-2b72-4c1f-8b51-9a13d2cfb485\") "
Oct 30 22:30:24.022766 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:24.022643    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a620edfb-2b72-4c1f-8b51-9a13d2cfb485-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a620edfb-2b72-4c1f-8b51-9a13d2cfb485-istiod-ca-cert\") pod \"a620edfb-2b72-4c1f-8b51-9a13d2cfb485\" (UID: \"a620edfb-2b72-4c1f-8b51-9a13d2cfb485\") "
Oct 30 22:30:24.022766 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:24.022681    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a620edfb-2b72-4c1f-8b51-9a13d2cfb485-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a620edfb-2b72-4c1f-8b51-9a13d2cfb485-istio-data\") pod \"a620edfb-2b72-4c1f-8b51-9a13d2cfb485\" (UID: \"a620edfb-2b72-4c1f-8b51-9a13d2cfb485\") "
Oct 30 22:30:24.022766 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:24.022745    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a620edfb-2b72-4c1f-8b51-9a13d2cfb485-istio-token\" (UniqueName: \"kubernetes.io/projected/a620edfb-2b72-4c1f-8b51-9a13d2cfb485-istio-token\") pod \"a620edfb-2b72-4c1f-8b51-9a13d2cfb485\" (UID: \"a620edfb-2b72-4c1f-8b51-9a13d2cfb485\") "
Oct 30 22:30:24.022966 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:24.022852    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a620edfb-2b72-4c1f-8b51-9a13d2cfb485-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a620edfb-2b72-4c1f-8b51-9a13d2cfb485-credential-socket\") pod \"a620edfb-2b72-4c1f-8b51-9a13d2cfb485\" (UID: \"a620edfb-2b72-4c1f-8b51-9a13d2cfb485\") "
Oct 30 22:30:24.022966 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:24.022869    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a620edfb-2b72-4c1f-8b51-9a13d2cfb485-workload-certs" pod "a620edfb-2b72-4c1f-8b51-9a13d2cfb485" (UID: "a620edfb-2b72-4c1f-8b51-9a13d2cfb485"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:24.022966 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:24.022911    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a620edfb-2b72-4c1f-8b51-9a13d2cfb485-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a620edfb-2b72-4c1f-8b51-9a13d2cfb485-istio-podinfo\") pod \"a620edfb-2b72-4c1f-8b51-9a13d2cfb485\" (UID: \"a620edfb-2b72-4c1f-8b51-9a13d2cfb485\") "
Oct 30 22:30:24.023175 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:24.023151    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a620edfb-2b72-4c1f-8b51-9a13d2cfb485-istio-data" pod "a620edfb-2b72-4c1f-8b51-9a13d2cfb485" (UID: "a620edfb-2b72-4c1f-8b51-9a13d2cfb485"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:24.023635 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:24.023242    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a620edfb-2b72-4c1f-8b51-9a13d2cfb485-istiod-ca-cert" pod "a620edfb-2b72-4c1f-8b51-9a13d2cfb485" (UID: "a620edfb-2b72-4c1f-8b51-9a13d2cfb485"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:24.023635 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:24.023296    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a620edfb-2b72-4c1f-8b51-9a13d2cfb485-kube-api-access-m45v4\" (UniqueName: \"kubernetes.io/projected/a620edfb-2b72-4c1f-8b51-9a13d2cfb485-kube-api-access-m45v4\") pod \"a620edfb-2b72-4c1f-8b51-9a13d2cfb485\" (UID: \"a620edfb-2b72-4c1f-8b51-9a13d2cfb485\") "
Oct 30 22:30:24.023635 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:24.023469    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a620edfb-2b72-4c1f-8b51-9a13d2cfb485-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a620edfb-2b72-4c1f-8b51-9a13d2cfb485-workload-socket\") pod \"a620edfb-2b72-4c1f-8b51-9a13d2cfb485\" (UID: \"a620edfb-2b72-4c1f-8b51-9a13d2cfb485\") "
Oct 30 22:30:24.023871 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:24.023664    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a620edfb-2b72-4c1f-8b51-9a13d2cfb485-credential-socket" pod "a620edfb-2b72-4c1f-8b51-9a13d2cfb485" (UID: "a620edfb-2b72-4c1f-8b51-9a13d2cfb485"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:24.024586 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:24.023933    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a620edfb-2b72-4c1f-8b51-9a13d2cfb485-workload-socket" pod "a620edfb-2b72-4c1f-8b51-9a13d2cfb485" (UID: "a620edfb-2b72-4c1f-8b51-9a13d2cfb485"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:24.024586 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:24.024495    4477 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a620edfb-2b72-4c1f-8b51-9a13d2cfb485-istiod-ca-cert\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:30:24.024586 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:24.024518    4477 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a620edfb-2b72-4c1f-8b51-9a13d2cfb485-istio-data\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:30:24.024586 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:24.024531    4477 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a620edfb-2b72-4c1f-8b51-9a13d2cfb485-credential-socket\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:30:24.024586 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:24.024546    4477 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a620edfb-2b72-4c1f-8b51-9a13d2cfb485-workload-socket\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:30:24.024586 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:24.024567    4477 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a620edfb-2b72-4c1f-8b51-9a13d2cfb485-workload-certs\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:30:24.026004 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:24.025981    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a620edfb-2b72-4c1f-8b51-9a13d2cfb485-istio-envoy" pod "a620edfb-2b72-4c1f-8b51-9a13d2cfb485" (UID: "a620edfb-2b72-4c1f-8b51-9a13d2cfb485"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:30:24.026149 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:24.026132    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a620edfb-2b72-4c1f-8b51-9a13d2cfb485-istio-token" pod "a620edfb-2b72-4c1f-8b51-9a13d2cfb485" (UID: "a620edfb-2b72-4c1f-8b51-9a13d2cfb485"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:24.026319 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:24.026289    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a620edfb-2b72-4c1f-8b51-9a13d2cfb485-istio-podinfo" pod "a620edfb-2b72-4c1f-8b51-9a13d2cfb485" (UID: "a620edfb-2b72-4c1f-8b51-9a13d2cfb485"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:30:24.027502 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:24.027475    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a620edfb-2b72-4c1f-8b51-9a13d2cfb485-kube-api-access-m45v4" pod "a620edfb-2b72-4c1f-8b51-9a13d2cfb485" (UID: "a620edfb-2b72-4c1f-8b51-9a13d2cfb485"). InnerVolumeSpecName "kube-api-access-m45v4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:24.126537 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:24.126509    4477 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a620edfb-2b72-4c1f-8b51-9a13d2cfb485-istio-podinfo\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:30:24.126666 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:24.126543    4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m45v4\" (UniqueName: \"kubernetes.io/projected/a620edfb-2b72-4c1f-8b51-9a13d2cfb485-kube-api-access-m45v4\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:30:24.126666 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:24.126558    4477 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a620edfb-2b72-4c1f-8b51-9a13d2cfb485-istio-envoy\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:30:24.126666 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:24.126571    4477 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a620edfb-2b72-4c1f-8b51-9a13d2cfb485-istio-token\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:30:24.322544 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:24.322504    4477 generic.go:356] "Generic (PLEG): container finished" podID="a620edfb-2b72-4c1f-8b51-9a13d2cfb485" containerID="601e77bb3bfbdfa9e4c4e7edd54bac35079d6bccbd64b290c5aee78520417df3" exitCode=0
Oct 30 22:30:24.322719 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:24.322559    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-47qsn"
Oct 30 22:30:24.322719 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:24.322589    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-47qsn" event={"ID":"a620edfb-2b72-4c1f-8b51-9a13d2cfb485","Type":"ContainerDied","Data":"601e77bb3bfbdfa9e4c4e7edd54bac35079d6bccbd64b290c5aee78520417df3"}
Oct 30 22:30:24.322719 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:24.322644    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-47qsn" event={"ID":"a620edfb-2b72-4c1f-8b51-9a13d2cfb485","Type":"ContainerDied","Data":"73175e746e718c26c13580fa71f1c299bff826a3df6f111658079e8a49ef22d8"}
Oct 30 22:30:24.322719 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:24.322666    4477 scope.go:122] "RemoveContainer" containerID="601e77bb3bfbdfa9e4c4e7edd54bac35079d6bccbd64b290c5aee78520417df3"
Oct 30 22:30:24.334157 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:24.334135    4477 scope.go:122] "RemoveContainer" containerID="601e77bb3bfbdfa9e4c4e7edd54bac35079d6bccbd64b290c5aee78520417df3"
Oct 30 22:30:24.334471 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:30:24.334448    4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"601e77bb3bfbdfa9e4c4e7edd54bac35079d6bccbd64b290c5aee78520417df3\": container with ID starting with 601e77bb3bfbdfa9e4c4e7edd54bac35079d6bccbd64b290c5aee78520417df3 not found: ID does not exist" containerID="601e77bb3bfbdfa9e4c4e7edd54bac35079d6bccbd64b290c5aee78520417df3"
Oct 30 22:30:24.334548 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:24.334484    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"601e77bb3bfbdfa9e4c4e7edd54bac35079d6bccbd64b290c5aee78520417df3"} err="failed to get container status \"601e77bb3bfbdfa9e4c4e7edd54bac35079d6bccbd64b290c5aee78520417df3\": rpc error: code = NotFound desc = could not find container \"601e77bb3bfbdfa9e4c4e7edd54bac35079d6bccbd64b290c5aee78520417df3\": container with ID starting with 601e77bb3bfbdfa9e4c4e7edd54bac35079d6bccbd64b290c5aee78520417df3 not found: ID does not exist"
Oct 30 22:30:24.404443 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:24.401949    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-47qsn"]
Oct 30 22:30:24.413112 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:24.412272    4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-d399fb07-openshift-default-6d54788969-47qsn"]
Oct 30 22:30:26.209756 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:26.206431    4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a620edfb-2b72-4c1f-8b51-9a13d2cfb485" path="/var/lib/kubelet/pods/a620edfb-2b72-4c1f-8b51-9a13d2cfb485/volumes"
Oct 30 22:30:26.275062 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:30:26.274070    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 22:30:27.356991 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:27.356958    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6bz92"]
Oct 30 22:30:29.462066 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:29.461660    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-de31dd-7d66978846-b464f"]
Oct 30 22:30:29.471108 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:29.470797    4477 cpu_manager.go:414] "containerMap: removing container" podUID="92d165b0-c272-4b3e-969d-ad9dfa6e84d9" containerName="istio-proxy"
Oct 30 22:30:29.471108 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:29.470827    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92d165b0-c272-4b3e-969d-ad9dfa6e84d9" containerName="istio-proxy"
Oct 30 22:30:29.471108 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:29.470845    4477 cpu_manager.go:414] "containerMap: removing container" podUID="ea5c4363-22dd-446e-a6df-1a0ce6497a87" containerName="cleanup"
Oct 30 22:30:29.471108 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:29.470853    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea5c4363-22dd-446e-a6df-1a0ce6497a87" containerName="cleanup"
Oct 30 22:30:29.471108 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:29.470873    4477 cpu_manager.go:414] "containerMap: removing container" podUID="852aabae-2489-4afe-ae9e-20d551cbf9a0" containerName="cleanup"
Oct 30 22:30:29.471108 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:29.470881    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="852aabae-2489-4afe-ae9e-20d551cbf9a0" containerName="cleanup"
Oct 30 22:30:29.471108 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:29.470897    4477 cpu_manager.go:414] "containerMap: removing container" podUID="1fb811da-4dc2-48bb-bec6-afb7a1a80ed2" containerName="istio-proxy"
Oct 30 22:30:29.471108 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:29.470905    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fb811da-4dc2-48bb-bec6-afb7a1a80ed2" containerName="istio-proxy"
Oct 30 22:30:29.471108 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:29.470926    4477 cpu_manager.go:414] "containerMap: removing container" podUID="a620edfb-2b72-4c1f-8b51-9a13d2cfb485" containerName="istio-proxy"
Oct 30 22:30:29.471108 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:29.470934    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a620edfb-2b72-4c1f-8b51-9a13d2cfb485" containerName="istio-proxy"
Oct 30 22:30:29.471108 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:29.470956    4477 cpu_manager.go:414] "containerMap: removing container" podUID="ea5c4363-22dd-446e-a6df-1a0ce6497a87" containerName="cleanup"
Oct 30 22:30:29.471108 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:29.470964    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea5c4363-22dd-446e-a6df-1a0ce6497a87" containerName="cleanup"
Oct 30 22:30:29.471108 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:29.470976    4477 cpu_manager.go:414] "containerMap: removing container" podUID="7237cb56-76fc-4438-b031-4c0eee080b7d" containerName="istio-proxy"
Oct 30 22:30:29.471108 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:29.470983    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7237cb56-76fc-4438-b031-4c0eee080b7d" containerName="istio-proxy"
Oct 30 22:30:29.471108 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:29.470994    4477 cpu_manager.go:414] "containerMap: removing container" podUID="2f5f8741-f496-46a2-b357-72c0e27ea2ae" containerName="istio-proxy"
Oct 30 22:30:29.471108 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:29.471002    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f5f8741-f496-46a2-b357-72c0e27ea2ae" containerName="istio-proxy"
Oct 30 22:30:29.471108 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:29.471014    4477 cpu_manager.go:414] "containerMap: removing container" podUID="ea5c4363-22dd-446e-a6df-1a0ce6497a87" containerName="cleanup"
Oct 30 22:30:29.471108 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:29.471022    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea5c4363-22dd-446e-a6df-1a0ce6497a87" containerName="cleanup"
Oct 30 22:30:29.471108 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:29.471052    4477 cpu_manager.go:414] "containerMap: removing container" podUID="852aabae-2489-4afe-ae9e-20d551cbf9a0" containerName="cleanup"
Oct 30 22:30:29.471108 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:29.471060    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="852aabae-2489-4afe-ae9e-20d551cbf9a0" containerName="cleanup"
Oct 30 22:30:29.471108 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:29.471088    4477 cpu_manager.go:414] "containerMap: removing container" podUID="8f11bbf1-c90c-4e20-a603-93dffec97774" containerName="maas-api"
Oct 30 22:30:29.471108 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:29.471095    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f11bbf1-c90c-4e20-a603-93dffec97774" containerName="maas-api"
Oct 30 22:30:29.471108 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:29.471115    4477 cpu_manager.go:414] "containerMap: removing container" podUID="720aa32b-9fc1-4013-bd84-a751a9f88fca" containerName="istio-proxy"
Oct 30 22:30:29.471108 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:29.471122    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="720aa32b-9fc1-4013-bd84-a751a9f88fca" containerName="istio-proxy"
Oct 30 22:30:29.471983 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:29.471315    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="852aabae-2489-4afe-ae9e-20d551cbf9a0" containerName="cleanup"
Oct 30 22:30:29.471983 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:29.471331    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f11bbf1-c90c-4e20-a603-93dffec97774" containerName="maas-api"
Oct 30 22:30:29.471983 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:29.471342    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="720aa32b-9fc1-4013-bd84-a751a9f88fca" containerName="istio-proxy"
Oct 30 22:30:29.471983 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:29.471355    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="92d165b0-c272-4b3e-969d-ad9dfa6e84d9" containerName="istio-proxy"
Oct 30 22:30:29.471983 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:29.471366    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea5c4363-22dd-446e-a6df-1a0ce6497a87" containerName="cleanup"
Oct 30 22:30:29.471983 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:29.471378    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fb811da-4dc2-48bb-bec6-afb7a1a80ed2" containerName="istio-proxy"
Oct 30 22:30:29.471983 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:29.471387    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="a620edfb-2b72-4c1f-8b51-9a13d2cfb485" containerName="istio-proxy"
Oct 30 22:30:29.471983 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:29.471396    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea5c4363-22dd-446e-a6df-1a0ce6497a87" containerName="cleanup"
Oct 30 22:30:29.471983 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:29.471406    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="7237cb56-76fc-4438-b031-4c0eee080b7d" containerName="istio-proxy"
Oct 30 22:30:29.471983 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:29.471416    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f5f8741-f496-46a2-b357-72c0e27ea2ae" containerName="istio-proxy"
Oct 30 22:30:29.471983 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:29.471426    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="852aabae-2489-4afe-ae9e-20d551cbf9a0" containerName="cleanup"
Oct 30 22:30:29.471983 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:29.471433    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea5c4363-22dd-446e-a6df-1a0ce6497a87" containerName="cleanup"
Oct 30 22:30:29.477259 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:29.477229    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-de31dd-7d66978846-b464f"
Oct 30 22:30:29.489014 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:29.488504    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-de31dd-7d66978846-b464f"]
Oct 30 22:30:29.508309 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:29.508284    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-de31dd-serving-cert\""
Oct 30 22:30:29.600515 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:29.600484    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2qvn\" (UniqueName: \"kubernetes.io/projected/0f677752-d3f8-4af6-b50d-ca2b63559ca8-kube-api-access-w2qvn\") pod \"maas-api-e2e-api-a-de31dd-7d66978846-b464f\" (UID: \"0f677752-d3f8-4af6-b50d-ca2b63559ca8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-de31dd-7d66978846-b464f"
Oct 30 22:30:29.600689 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:29.600526    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f677752-d3f8-4af6-b50d-ca2b63559ca8-maas-api-tls\") pod \"maas-api-e2e-api-a-de31dd-7d66978846-b464f\" (UID: \"0f677752-d3f8-4af6-b50d-ca2b63559ca8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-de31dd-7d66978846-b464f"
Oct 30 22:30:29.600689 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:29.600545    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f677752-d3f8-4af6-b50d-ca2b63559ca8-openshift-service-ca\") pod \"maas-api-e2e-api-a-de31dd-7d66978846-b464f\" (UID: \"0f677752-d3f8-4af6-b50d-ca2b63559ca8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-de31dd-7d66978846-b464f"
Oct 30 22:30:29.600774 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:29.600739    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f677752-d3f8-4af6-b50d-ca2b63559ca8-metrics-certs\") pod \"maas-api-e2e-api-a-de31dd-7d66978846-b464f\" (UID: \"0f677752-d3f8-4af6-b50d-ca2b63559ca8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-de31dd-7d66978846-b464f"
Oct 30 22:30:29.702598 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:29.702562    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f677752-d3f8-4af6-b50d-ca2b63559ca8-metrics-certs\") pod \"maas-api-e2e-api-a-de31dd-7d66978846-b464f\" (UID: \"0f677752-d3f8-4af6-b50d-ca2b63559ca8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-de31dd-7d66978846-b464f"
Oct 30 22:30:29.702781 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:29.702716    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2qvn\" (UniqueName: \"kubernetes.io/projected/0f677752-d3f8-4af6-b50d-ca2b63559ca8-kube-api-access-w2qvn\") pod \"maas-api-e2e-api-a-de31dd-7d66978846-b464f\" (UID: \"0f677752-d3f8-4af6-b50d-ca2b63559ca8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-de31dd-7d66978846-b464f"
Oct 30 22:30:29.702781 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:29.702746    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f677752-d3f8-4af6-b50d-ca2b63559ca8-maas-api-tls\") pod \"maas-api-e2e-api-a-de31dd-7d66978846-b464f\" (UID: \"0f677752-d3f8-4af6-b50d-ca2b63559ca8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-de31dd-7d66978846-b464f"
Oct 30 22:30:29.702781 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:29.702761    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f677752-d3f8-4af6-b50d-ca2b63559ca8-openshift-service-ca\") pod \"maas-api-e2e-api-a-de31dd-7d66978846-b464f\" (UID: \"0f677752-d3f8-4af6-b50d-ca2b63559ca8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-de31dd-7d66978846-b464f"
Oct 30 22:30:29.702956 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:30:29.702873    4477 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-de31dd-serving-cert: secret "maas-api-e2e-api-a-de31dd-serving-cert" not found
Oct 30 22:30:29.703013 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:30:29.702960    4477 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f677752-d3f8-4af6-b50d-ca2b63559ca8-maas-api-tls podName:0f677752-d3f8-4af6-b50d-ca2b63559ca8 nodeName:}" failed. No retries permitted until 2026-10-30 22:30:30.202939338 +0000 UTC m=+1414.128865536 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0f677752-d3f8-4af6-b50d-ca2b63559ca8-maas-api-tls") pod "maas-api-e2e-api-a-de31dd-7d66978846-b464f" (UID: "0f677752-d3f8-4af6-b50d-ca2b63559ca8") : secret "maas-api-e2e-api-a-de31dd-serving-cert" not found
Oct 30 22:30:29.703515 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:29.703485    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f677752-d3f8-4af6-b50d-ca2b63559ca8-openshift-service-ca\") pod \"maas-api-e2e-api-a-de31dd-7d66978846-b464f\" (UID: \"0f677752-d3f8-4af6-b50d-ca2b63559ca8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-de31dd-7d66978846-b464f"
Oct 30 22:30:29.705645 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:29.705617    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f677752-d3f8-4af6-b50d-ca2b63559ca8-metrics-certs\") pod \"maas-api-e2e-api-a-de31dd-7d66978846-b464f\" (UID: \"0f677752-d3f8-4af6-b50d-ca2b63559ca8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-de31dd-7d66978846-b464f"
Oct 30 22:30:29.720682 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:29.720627    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2qvn\" (UniqueName: \"kubernetes.io/projected/0f677752-d3f8-4af6-b50d-ca2b63559ca8-kube-api-access-w2qvn\") pod \"maas-api-e2e-api-a-de31dd-7d66978846-b464f\" (UID: \"0f677752-d3f8-4af6-b50d-ca2b63559ca8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-de31dd-7d66978846-b464f"
Oct 30 22:30:30.211242 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:30.211208    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f677752-d3f8-4af6-b50d-ca2b63559ca8-maas-api-tls\") pod \"maas-api-e2e-api-a-de31dd-7d66978846-b464f\" (UID: \"0f677752-d3f8-4af6-b50d-ca2b63559ca8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-de31dd-7d66978846-b464f"
Oct 30 22:30:30.213728 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:30.213700    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f677752-d3f8-4af6-b50d-ca2b63559ca8-maas-api-tls\") pod \"maas-api-e2e-api-a-de31dd-7d66978846-b464f\" (UID: \"0f677752-d3f8-4af6-b50d-ca2b63559ca8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-de31dd-7d66978846-b464f"
Oct 30 22:30:30.417210 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:30.417174    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-de31dd-7d66978846-b464f"
Oct 30 22:30:30.734053 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:30.734005    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-de31dd-7d66978846-b464f"]
Oct 30 22:30:30.737931 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:30:30.737896    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f677752_d3f8_4af6_b50d_ca2b63559ca8.slice/crio-e12534318403ec9ec763c33ccd216dacc73744dda87482c1b3969b8eb9cfec30 WatchSource:0}: Error finding container e12534318403ec9ec763c33ccd216dacc73744dda87482c1b3969b8eb9cfec30: Status 404 returned error can't find the container with id e12534318403ec9ec763c33ccd216dacc73744dda87482c1b3969b8eb9cfec30
Oct 30 22:30:31.376474 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:31.376438    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-de31dd-7d66978846-b464f" event={"ID":"0f677752-d3f8-4af6-b50d-ca2b63559ca8","Type":"ContainerStarted","Data":"ea8ee8d06c436c491865df25e3c34c352401f1cb455458a3d18c13c53dd0bac3"}
Oct 30 22:30:31.376642 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:31.376505    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-de31dd-7d66978846-b464f" event={"ID":"0f677752-d3f8-4af6-b50d-ca2b63559ca8","Type":"ContainerStarted","Data":"e12534318403ec9ec763c33ccd216dacc73744dda87482c1b3969b8eb9cfec30"}
Oct 30 22:30:31.376642 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:31.376558    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-de31dd-7d66978846-b464f"
Oct 30 22:30:31.420806 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:31.420753    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-de31dd-7d66978846-b464f" podStartSLOduration=2.06307473 podStartE2EDuration="2.420739668s" totalImagesPullingTime="357.664938ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:31.41594915 +0000 UTC m=+1415.341875386" watchObservedRunningTime="2026-10-30 22:30:31.420739668 +0000 UTC m=+1415.346665906"
Oct 30 22:30:37.389426 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:37.389397    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-de31dd-7d66978846-b464f"
Oct 30 22:30:40.574177 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:40.574142    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-9nsmf"]
Oct 30 22:30:40.575131 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:40.575111    4477 cpu_manager.go:414] "containerMap: removing container" podUID="852aabae-2489-4afe-ae9e-20d551cbf9a0" containerName="cleanup"
Oct 30 22:30:40.575174 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:40.575135    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="852aabae-2489-4afe-ae9e-20d551cbf9a0" containerName="cleanup"
Oct 30 22:30:40.575408 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:40.575393    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="852aabae-2489-4afe-ae9e-20d551cbf9a0" containerName="cleanup"
Oct 30 22:30:40.579608 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:40.579590    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-9nsmf"
Oct 30 22:30:40.608118 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:40.608093    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-370ec2-openshift-default-dockercfg-lcw59\""
Oct 30 22:30:40.626160 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:40.626122    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-9nsmf"]
Oct 30 22:30:40.664031 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:40.664004    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abdba985-1e18-4dc3-b3bb-2456a71609f2-workload-socket\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-9nsmf\" (UID: \"abdba985-1e18-4dc3-b3bb-2456a71609f2\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-9nsmf"
Oct 30 22:30:40.664180 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:40.664164    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7nqj\" (UniqueName: \"kubernetes.io/projected/abdba985-1e18-4dc3-b3bb-2456a71609f2-kube-api-access-k7nqj\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-9nsmf\" (UID: \"abdba985-1e18-4dc3-b3bb-2456a71609f2\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-9nsmf"
Oct 30 22:30:40.664234 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:40.664223    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abdba985-1e18-4dc3-b3bb-2456a71609f2-credential-socket\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-9nsmf\" (UID: \"abdba985-1e18-4dc3-b3bb-2456a71609f2\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-9nsmf"
Oct 30 22:30:40.664303 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:40.664283    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abdba985-1e18-4dc3-b3bb-2456a71609f2-istio-podinfo\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-9nsmf\" (UID: \"abdba985-1e18-4dc3-b3bb-2456a71609f2\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-9nsmf"
Oct 30 22:30:40.664397 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:40.664384    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abdba985-1e18-4dc3-b3bb-2456a71609f2-istiod-ca-cert\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-9nsmf\" (UID: \"abdba985-1e18-4dc3-b3bb-2456a71609f2\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-9nsmf"
Oct 30 22:30:40.664463 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:40.664452    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abdba985-1e18-4dc3-b3bb-2456a71609f2-istio-envoy\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-9nsmf\" (UID: \"abdba985-1e18-4dc3-b3bb-2456a71609f2\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-9nsmf"
Oct 30 22:30:40.664530 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:40.664513    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abdba985-1e18-4dc3-b3bb-2456a71609f2-istio-token\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-9nsmf\" (UID: \"abdba985-1e18-4dc3-b3bb-2456a71609f2\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-9nsmf"
Oct 30 22:30:40.664626 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:40.664606    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abdba985-1e18-4dc3-b3bb-2456a71609f2-workload-certs\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-9nsmf\" (UID: \"abdba985-1e18-4dc3-b3bb-2456a71609f2\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-9nsmf"
Oct 30 22:30:40.664670 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:40.664658    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abdba985-1e18-4dc3-b3bb-2456a71609f2-istio-data\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-9nsmf\" (UID: \"abdba985-1e18-4dc3-b3bb-2456a71609f2\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-9nsmf"
Oct 30 22:30:40.766371 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:40.766343    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7nqj\" (UniqueName: \"kubernetes.io/projected/abdba985-1e18-4dc3-b3bb-2456a71609f2-kube-api-access-k7nqj\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-9nsmf\" (UID: \"abdba985-1e18-4dc3-b3bb-2456a71609f2\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-9nsmf"
Oct 30 22:30:40.766490 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:40.766402    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abdba985-1e18-4dc3-b3bb-2456a71609f2-credential-socket\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-9nsmf\" (UID: \"abdba985-1e18-4dc3-b3bb-2456a71609f2\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-9nsmf"
Oct 30 22:30:40.766490 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:40.766428    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abdba985-1e18-4dc3-b3bb-2456a71609f2-istio-podinfo\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-9nsmf\" (UID: \"abdba985-1e18-4dc3-b3bb-2456a71609f2\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-9nsmf"
Oct 30 22:30:40.766490 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:40.766455    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abdba985-1e18-4dc3-b3bb-2456a71609f2-istiod-ca-cert\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-9nsmf\" (UID: \"abdba985-1e18-4dc3-b3bb-2456a71609f2\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-9nsmf"
Oct 30 22:30:40.766694 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:40.766672    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abdba985-1e18-4dc3-b3bb-2456a71609f2-istio-envoy\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-9nsmf\" (UID: \"abdba985-1e18-4dc3-b3bb-2456a71609f2\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-9nsmf"
Oct 30 22:30:40.766764 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:40.766726    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abdba985-1e18-4dc3-b3bb-2456a71609f2-istio-token\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-9nsmf\" (UID: \"abdba985-1e18-4dc3-b3bb-2456a71609f2\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-9nsmf"
Oct 30 22:30:40.766816 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:40.766785    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abdba985-1e18-4dc3-b3bb-2456a71609f2-workload-certs\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-9nsmf\" (UID: \"abdba985-1e18-4dc3-b3bb-2456a71609f2\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-9nsmf"
Oct 30 22:30:40.766867 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:40.766827    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abdba985-1e18-4dc3-b3bb-2456a71609f2-credential-socket\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-9nsmf\" (UID: \"abdba985-1e18-4dc3-b3bb-2456a71609f2\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-9nsmf"
Oct 30 22:30:40.766867 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:40.766830    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abdba985-1e18-4dc3-b3bb-2456a71609f2-istio-data\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-9nsmf\" (UID: \"abdba985-1e18-4dc3-b3bb-2456a71609f2\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-9nsmf"
Oct 30 22:30:40.766973 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:40.766931    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abdba985-1e18-4dc3-b3bb-2456a71609f2-workload-socket\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-9nsmf\" (UID: \"abdba985-1e18-4dc3-b3bb-2456a71609f2\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-9nsmf"
Oct 30 22:30:40.767190 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:40.767166    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abdba985-1e18-4dc3-b3bb-2456a71609f2-istio-data\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-9nsmf\" (UID: \"abdba985-1e18-4dc3-b3bb-2456a71609f2\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-9nsmf"
Oct 30 22:30:40.767290 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:40.767199    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abdba985-1e18-4dc3-b3bb-2456a71609f2-workload-certs\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-9nsmf\" (UID: \"abdba985-1e18-4dc3-b3bb-2456a71609f2\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-9nsmf"
Oct 30 22:30:40.767290 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:40.767209    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abdba985-1e18-4dc3-b3bb-2456a71609f2-istiod-ca-cert\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-9nsmf\" (UID: \"abdba985-1e18-4dc3-b3bb-2456a71609f2\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-9nsmf"
Oct 30 22:30:40.767290 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:40.767285    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abdba985-1e18-4dc3-b3bb-2456a71609f2-workload-socket\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-9nsmf\" (UID: \"abdba985-1e18-4dc3-b3bb-2456a71609f2\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-9nsmf"
Oct 30 22:30:40.769147 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:40.769120    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abdba985-1e18-4dc3-b3bb-2456a71609f2-istio-podinfo\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-9nsmf\" (UID: \"abdba985-1e18-4dc3-b3bb-2456a71609f2\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-9nsmf"
Oct 30 22:30:40.769269 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:40.769193    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abdba985-1e18-4dc3-b3bb-2456a71609f2-istio-envoy\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-9nsmf\" (UID: \"abdba985-1e18-4dc3-b3bb-2456a71609f2\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-9nsmf"
Oct 30 22:30:40.784471 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:40.784446    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abdba985-1e18-4dc3-b3bb-2456a71609f2-istio-token\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-9nsmf\" (UID: \"abdba985-1e18-4dc3-b3bb-2456a71609f2\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-9nsmf"
Oct 30 22:30:40.784565 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:40.784466    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7nqj\" (UniqueName: \"kubernetes.io/projected/abdba985-1e18-4dc3-b3bb-2456a71609f2-kube-api-access-k7nqj\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-9nsmf\" (UID: \"abdba985-1e18-4dc3-b3bb-2456a71609f2\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-9nsmf"
Oct 30 22:30:40.895077 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:40.895028    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-9nsmf"
Oct 30 22:30:41.210976 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:41.210948    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-9nsmf"]
Oct 30 22:30:41.213714 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:30:41.213687    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabdba985_1e18_4dc3_b3bb_2456a71609f2.slice/crio-0a515fed01402c1da4b69b1501e10f64ce488c8318e4210d4404ed74409de319 WatchSource:0}: Error finding container 0a515fed01402c1da4b69b1501e10f64ce488c8318e4210d4404ed74409de319: Status 404 returned error can't find the container with id 0a515fed01402c1da4b69b1501e10f64ce488c8318e4210d4404ed74409de319
Oct 30 22:30:41.216749 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:41.216707    4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:30:41.216836 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:41.216793    4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:30:41.216873 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:41.216836    4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:30:41.451078 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:41.448014    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-9nsmf" event={"ID":"abdba985-1e18-4dc3-b3bb-2456a71609f2","Type":"ContainerStarted","Data":"824f67ed98229f197007d8bddfb896de48cc707fd203edcda71ab8f591ee486f"}
Oct 30 22:30:41.451078 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:41.448117    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-9nsmf" event={"ID":"abdba985-1e18-4dc3-b3bb-2456a71609f2","Type":"ContainerStarted","Data":"0a515fed01402c1da4b69b1501e10f64ce488c8318e4210d4404ed74409de319"}
Oct 30 22:30:41.495122 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:41.494591    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-9nsmf" podStartSLOduration=1.494577397 podStartE2EDuration="1.494577397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:41.488575527 +0000 UTC m=+1425.414501726" watchObservedRunningTime="2026-10-30 22:30:41.494577397 +0000 UTC m=+1425.420503653"
Oct 30 22:30:41.907184 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:41.907144    4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-9nsmf"
Oct 30 22:30:42.897035 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:42.896989    4477 patch_prober.go:29] interesting pod/e2e-api-a-370ec2-openshift-default-6fccc4f697-9nsmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Oct 30 22:30:42.897228 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:42.897082    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-9nsmf" podUID="abdba985-1e18-4dc3-b3bb-2456a71609f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Oct 30 22:30:43.895499 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:43.895466    4477 patch_prober.go:29] interesting pod/e2e-api-a-370ec2-openshift-default-6fccc4f697-9nsmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Oct 30 22:30:43.895915 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:43.895522    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-9nsmf" podUID="abdba985-1e18-4dc3-b3bb-2456a71609f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Oct 30 22:30:44.899463 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:44.899422    4477 patch_prober.go:29] interesting pod/e2e-api-a-370ec2-openshift-default-6fccc4f697-9nsmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Oct 30 22:30:44.900095 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:44.900061    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-9nsmf" podUID="abdba985-1e18-4dc3-b3bb-2456a71609f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Oct 30 22:30:45.895690 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:45.895651    4477 patch_prober.go:29] interesting pod/e2e-api-a-370ec2-openshift-default-6fccc4f697-9nsmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Oct 30 22:30:45.896127 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:45.896091    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-9nsmf" podUID="abdba985-1e18-4dc3-b3bb-2456a71609f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Oct 30 22:30:46.905452 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:46.905413    4477 patch_prober.go:29] interesting pod/e2e-api-a-370ec2-openshift-default-6fccc4f697-9nsmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Oct 30 22:30:46.918638 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:46.918596    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-9nsmf" podUID="abdba985-1e18-4dc3-b3bb-2456a71609f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Oct 30 22:30:47.900265 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:47.900234    4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-9nsmf"
Oct 30 22:30:47.900534 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:47.900510    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-9nsmf"
Oct 30 22:30:47.901269 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:47.901250    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-9nsmf"
Oct 30 22:30:56.270799 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:30:56.270757    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 22:30:56.640176 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:56.640138    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7278-b18f20"]
Oct 30 22:30:56.645290 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:56.645268    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7278-b18f20"
Oct 30 22:30:56.660477 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:56.660432    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7278-b18f20"]
Oct 30 22:30:56.711726 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:56.711684    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wmwh\" (UniqueName: \"kubernetes.io/projected/36760e39-2943-4beb-b833-0a70dfa23921-kube-api-access-7wmwh\") pod \"test-curl-7278-b18f20\" (UID: \"36760e39-2943-4beb-b833-0a70dfa23921\") " pod="opendatahub/test-curl-7278-b18f20"
Oct 30 22:30:56.714813 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:56.714789    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-xbw52\""
Oct 30 22:30:56.813029 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:56.812993    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wmwh\" (UniqueName: \"kubernetes.io/projected/36760e39-2943-4beb-b833-0a70dfa23921-kube-api-access-7wmwh\") pod \"test-curl-7278-b18f20\" (UID: \"36760e39-2943-4beb-b833-0a70dfa23921\") " pod="opendatahub/test-curl-7278-b18f20"
Oct 30 22:30:56.831219 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:56.831191    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wmwh\" (UniqueName: \"kubernetes.io/projected/36760e39-2943-4beb-b833-0a70dfa23921-kube-api-access-7wmwh\") pod \"test-curl-7278-b18f20\" (UID: \"36760e39-2943-4beb-b833-0a70dfa23921\") " pod="opendatahub/test-curl-7278-b18f20"
Oct 30 22:30:56.961583 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:56.961518    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7278-b18f20"
Oct 30 22:30:57.283597 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:57.283527    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7278-b18f20"]
Oct 30 22:30:57.286957 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:30:57.286917    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod36760e39_2943_4beb_b833_0a70dfa23921.slice/crio-bea915096c3cee9d06c0b6fd11806e2c1bbd805356d9fea9e6a8e8f6aa9981ba WatchSource:0}: Error finding container bea915096c3cee9d06c0b6fd11806e2c1bbd805356d9fea9e6a8e8f6aa9981ba: Status 404 returned error can't find the container with id bea915096c3cee9d06c0b6fd11806e2c1bbd805356d9fea9e6a8e8f6aa9981ba
Oct 30 22:30:57.655927 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:57.655895    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7278-b18f20" event={"ID":"36760e39-2943-4beb-b833-0a70dfa23921","Type":"ContainerStarted","Data":"8b5ee5bbe8a713a24f4e28044aa462492595b8c39c759c631ab5c249178b9362"}
Oct 30 22:30:57.656099 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:57.655938    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7278-b18f20" event={"ID":"36760e39-2943-4beb-b833-0a70dfa23921","Type":"ContainerStarted","Data":"bea915096c3cee9d06c0b6fd11806e2c1bbd805356d9fea9e6a8e8f6aa9981ba"}
Oct 30 22:30:57.691618 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:57.691571    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7278-b18f20" podStartSLOduration=1.69155859 podStartE2EDuration="1.69155859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:57.690225744 +0000 UTC m=+1441.616151952" watchObservedRunningTime="2026-10-30 22:30:57.69155859 +0000 UTC m=+1441.617484795"
Oct 30 22:30:58.518749 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:58.518718    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7278-b18f20"]
Oct 30 22:30:58.522792 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:58.522760    4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7278-b18f20"]
Oct 30 22:30:59.676104 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:30:59.671245    4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7278-b18f20" podUID="36760e39-2943-4beb-b833-0a70dfa23921" containerName="test-curl-7278-b18f20" containerID="cri-o://8b5ee5bbe8a713a24f4e28044aa462492595b8c39c759c631ab5c249178b9362" gracePeriod=2
Oct 30 22:30:59.682990 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:30:59.682957    4477 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7278-b18f20\" is forbidden: User \"system:node:ip-10-0-130-77.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-77.ec2.internal' and this object" podUID="36760e39-2943-4beb-b833-0a70dfa23921" pod="opendatahub/test-curl-7278-b18f20"
Oct 30 22:31:01.878161 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:01.878137    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7278-b18f20"
Oct 30 22:31:01.986000 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:01.985929    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36760e39-2943-4beb-b833-0a70dfa23921-kube-api-access-7wmwh\" (UniqueName: \"kubernetes.io/projected/36760e39-2943-4beb-b833-0a70dfa23921-kube-api-access-7wmwh\") pod \"36760e39-2943-4beb-b833-0a70dfa23921\" (UID: \"36760e39-2943-4beb-b833-0a70dfa23921\") "
Oct 30 22:31:01.987917 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:01.987896    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36760e39-2943-4beb-b833-0a70dfa23921-kube-api-access-7wmwh" pod "36760e39-2943-4beb-b833-0a70dfa23921" (UID: "36760e39-2943-4beb-b833-0a70dfa23921"). InnerVolumeSpecName "kube-api-access-7wmwh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:31:02.087847 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:02.087819    4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wmwh\" (UniqueName: \"kubernetes.io/projected/36760e39-2943-4beb-b833-0a70dfa23921-kube-api-access-7wmwh\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:31:02.195248 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:02.195217    4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36760e39-2943-4beb-b833-0a70dfa23921" path="/var/lib/kubelet/pods/36760e39-2943-4beb-b833-0a70dfa23921/volumes"
Oct 30 22:31:02.692184 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:02.692153    4477 generic.go:356] "Generic (PLEG): container finished" podID="36760e39-2943-4beb-b833-0a70dfa23921" containerID="8b5ee5bbe8a713a24f4e28044aa462492595b8c39c759c631ab5c249178b9362" exitCode=137
Oct 30 22:31:02.692350 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:02.692198    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7278-b18f20"
Oct 30 22:31:02.692350 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:02.692253    4477 scope.go:122] "RemoveContainer" containerID="8b5ee5bbe8a713a24f4e28044aa462492595b8c39c759c631ab5c249178b9362"
Oct 30 22:31:02.704022 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:02.704006    4477 scope.go:122] "RemoveContainer" containerID="8b5ee5bbe8a713a24f4e28044aa462492595b8c39c759c631ab5c249178b9362"
Oct 30 22:31:02.704313 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:31:02.704291    4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b5ee5bbe8a713a24f4e28044aa462492595b8c39c759c631ab5c249178b9362\": container with ID starting with 8b5ee5bbe8a713a24f4e28044aa462492595b8c39c759c631ab5c249178b9362 not found: ID does not exist" containerID="8b5ee5bbe8a713a24f4e28044aa462492595b8c39c759c631ab5c249178b9362"
Oct 30 22:31:02.704357 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:02.704322    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b5ee5bbe8a713a24f4e28044aa462492595b8c39c759c631ab5c249178b9362"} err="failed to get container status \"8b5ee5bbe8a713a24f4e28044aa462492595b8c39c759c631ab5c249178b9362\": rpc error: code = NotFound desc = could not find container \"8b5ee5bbe8a713a24f4e28044aa462492595b8c39c759c631ab5c249178b9362\": container with ID starting with 8b5ee5bbe8a713a24f4e28044aa462492595b8c39c759c631ab5c249178b9362 not found: ID does not exist"
Oct 30 22:31:06.453203 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:06.453171    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-30edd7-5ff8d79b9d-2lqtr"]
Oct 30 22:31:06.453805 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:06.453791    4477 cpu_manager.go:414] "containerMap: removing container" podUID="36760e39-2943-4beb-b833-0a70dfa23921" containerName="test-curl-7278-b18f20"
Oct 30 22:31:06.453849 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:06.453806    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36760e39-2943-4beb-b833-0a70dfa23921" containerName="test-curl-7278-b18f20"
Oct 30 22:31:06.453924 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:06.453915    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="36760e39-2943-4beb-b833-0a70dfa23921" containerName="test-curl-7278-b18f20"
Oct 30 22:31:06.458987 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:06.458969    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-30edd7-5ff8d79b9d-2lqtr"
Oct 30 22:31:06.482516 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:06.482489    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-30edd7-serving-cert\""
Oct 30 22:31:06.489086 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:06.488003    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-30edd7-5ff8d79b9d-2lqtr"]
Oct 30 22:31:06.543995 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:06.543968    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2577\" (UniqueName: \"kubernetes.io/projected/d278732b-d633-4a53-b3ce-1a3fc9ad6790-kube-api-access-k2577\") pod \"maas-api-e2e-api-a-30edd7-5ff8d79b9d-2lqtr\" (UID: \"d278732b-d633-4a53-b3ce-1a3fc9ad6790\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-30edd7-5ff8d79b9d-2lqtr"
Oct 30 22:31:06.544164 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:06.544004    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d278732b-d633-4a53-b3ce-1a3fc9ad6790-metrics-certs\") pod \"maas-api-e2e-api-a-30edd7-5ff8d79b9d-2lqtr\" (UID: \"d278732b-d633-4a53-b3ce-1a3fc9ad6790\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-30edd7-5ff8d79b9d-2lqtr"
Oct 30 22:31:06.544164 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:06.544109    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d278732b-d633-4a53-b3ce-1a3fc9ad6790-maas-api-tls\") pod \"maas-api-e2e-api-a-30edd7-5ff8d79b9d-2lqtr\" (UID: \"d278732b-d633-4a53-b3ce-1a3fc9ad6790\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-30edd7-5ff8d79b9d-2lqtr"
Oct 30 22:31:06.544337 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:06.544323    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d278732b-d633-4a53-b3ce-1a3fc9ad6790-openshift-service-ca\") pod \"maas-api-e2e-api-a-30edd7-5ff8d79b9d-2lqtr\" (UID: \"d278732b-d633-4a53-b3ce-1a3fc9ad6790\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-30edd7-5ff8d79b9d-2lqtr"
Oct 30 22:31:06.645571 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:06.645542    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d278732b-d633-4a53-b3ce-1a3fc9ad6790-openshift-service-ca\") pod \"maas-api-e2e-api-a-30edd7-5ff8d79b9d-2lqtr\" (UID: \"d278732b-d633-4a53-b3ce-1a3fc9ad6790\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-30edd7-5ff8d79b9d-2lqtr"
Oct 30 22:31:06.645818 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:06.645794    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2577\" (UniqueName: \"kubernetes.io/projected/d278732b-d633-4a53-b3ce-1a3fc9ad6790-kube-api-access-k2577\") pod \"maas-api-e2e-api-a-30edd7-5ff8d79b9d-2lqtr\" (UID: \"d278732b-d633-4a53-b3ce-1a3fc9ad6790\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-30edd7-5ff8d79b9d-2lqtr"
Oct 30 22:31:06.645934 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:06.645838    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d278732b-d633-4a53-b3ce-1a3fc9ad6790-metrics-certs\") pod \"maas-api-e2e-api-a-30edd7-5ff8d79b9d-2lqtr\" (UID: \"d278732b-d633-4a53-b3ce-1a3fc9ad6790\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-30edd7-5ff8d79b9d-2lqtr"
Oct 30 22:31:06.645934 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:06.645871    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d278732b-d633-4a53-b3ce-1a3fc9ad6790-maas-api-tls\") pod \"maas-api-e2e-api-a-30edd7-5ff8d79b9d-2lqtr\" (UID: \"d278732b-d633-4a53-b3ce-1a3fc9ad6790\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-30edd7-5ff8d79b9d-2lqtr"
Oct 30 22:31:06.646161 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:31:06.646094    4477 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-30edd7-serving-cert: secret "maas-api-e2e-api-a-30edd7-serving-cert" not found
Oct 30 22:31:06.646203 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:31:06.646162    4477 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d278732b-d633-4a53-b3ce-1a3fc9ad6790-maas-api-tls podName:d278732b-d633-4a53-b3ce-1a3fc9ad6790 nodeName:}" failed. No retries permitted until 2026-10-30 22:31:07.146147836 +0000 UTC m=+1451.072074034 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d278732b-d633-4a53-b3ce-1a3fc9ad6790-maas-api-tls") pod "maas-api-e2e-api-a-30edd7-5ff8d79b9d-2lqtr" (UID: "d278732b-d633-4a53-b3ce-1a3fc9ad6790") : secret "maas-api-e2e-api-a-30edd7-serving-cert" not found
Oct 30 22:31:06.646344 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:06.646317    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d278732b-d633-4a53-b3ce-1a3fc9ad6790-openshift-service-ca\") pod \"maas-api-e2e-api-a-30edd7-5ff8d79b9d-2lqtr\" (UID: \"d278732b-d633-4a53-b3ce-1a3fc9ad6790\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-30edd7-5ff8d79b9d-2lqtr"
Oct 30 22:31:06.648667 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:06.648639    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d278732b-d633-4a53-b3ce-1a3fc9ad6790-metrics-certs\") pod \"maas-api-e2e-api-a-30edd7-5ff8d79b9d-2lqtr\" (UID: \"d278732b-d633-4a53-b3ce-1a3fc9ad6790\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-30edd7-5ff8d79b9d-2lqtr"
Oct 30 22:31:06.664732 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:06.664703    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2577\" (UniqueName: \"kubernetes.io/projected/d278732b-d633-4a53-b3ce-1a3fc9ad6790-kube-api-access-k2577\") pod \"maas-api-e2e-api-a-30edd7-5ff8d79b9d-2lqtr\" (UID: \"d278732b-d633-4a53-b3ce-1a3fc9ad6790\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-30edd7-5ff8d79b9d-2lqtr"
Oct 30 22:31:07.153555 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:07.153511    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d278732b-d633-4a53-b3ce-1a3fc9ad6790-maas-api-tls\") pod \"maas-api-e2e-api-a-30edd7-5ff8d79b9d-2lqtr\" (UID: \"d278732b-d633-4a53-b3ce-1a3fc9ad6790\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-30edd7-5ff8d79b9d-2lqtr"
Oct 30 22:31:07.156260 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:07.156235    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d278732b-d633-4a53-b3ce-1a3fc9ad6790-maas-api-tls\") pod \"maas-api-e2e-api-a-30edd7-5ff8d79b9d-2lqtr\" (UID: \"d278732b-d633-4a53-b3ce-1a3fc9ad6790\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-30edd7-5ff8d79b9d-2lqtr"
Oct 30 22:31:07.373093 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:07.373024    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-30edd7-5ff8d79b9d-2lqtr"
Oct 30 22:31:07.703206 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:07.703170    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-30edd7-5ff8d79b9d-2lqtr"]
Oct 30 22:31:07.706850 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:31:07.706823    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd278732b_d633_4a53_b3ce_1a3fc9ad6790.slice/crio-721756434bca1d0b39fefc84c214134f6f734419407cf2c76eef8e9696708f08 WatchSource:0}: Error finding container 721756434bca1d0b39fefc84c214134f6f734419407cf2c76eef8e9696708f08: Status 404 returned error can't find the container with id 721756434bca1d0b39fefc84c214134f6f734419407cf2c76eef8e9696708f08
Oct 30 22:31:07.729668 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:07.729644    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-30edd7-5ff8d79b9d-2lqtr" event={"ID":"d278732b-d633-4a53-b3ce-1a3fc9ad6790","Type":"ContainerStarted","Data":"721756434bca1d0b39fefc84c214134f6f734419407cf2c76eef8e9696708f08"}
Oct 30 22:31:08.368922 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:08.368842    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6bz92"]
Oct 30 22:31:08.738747 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:08.738707    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-30edd7-5ff8d79b9d-2lqtr" event={"ID":"d278732b-d633-4a53-b3ce-1a3fc9ad6790","Type":"ContainerStarted","Data":"56c1075fdef60130d43328bf6e9b32f1a45679617d5fe12d18b4a8d7acd38ff0"}
Oct 30 22:31:08.739222 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:08.738903    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-30edd7-5ff8d79b9d-2lqtr"
Oct 30 22:31:08.781603 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:08.781553    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-30edd7-5ff8d79b9d-2lqtr" podStartSLOduration=2.379558047 podStartE2EDuration="2.781540159s" totalImagesPullingTime="401.982112ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:31:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:31:08.777679683 +0000 UTC m=+1452.703605916" watchObservedRunningTime="2026-10-30 22:31:08.781540159 +0000 UTC m=+1452.707466365"
Oct 30 22:31:09.166515 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:09.166481    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-z7ggc"]
Oct 30 22:31:09.166766 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:09.166699    4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-z7ggc" podUID="5779d15a-20b4-4ec0-ac5e-b8bdfa89096d" containerName="istio-proxy" containerID="cri-o://2444cbe557ac681373e56ac69c36d693830c7a572f5cd296ec1b7c95dbaebe32" gracePeriod=30
Oct 30 22:31:09.723976 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:09.723940    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-sqxvr"]
Oct 30 22:31:09.724197 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:09.724173    4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-sqxvr" podUID="909907e9-319a-48dc-90d6-735cee614217" containerName="istio-proxy" containerID="cri-o://5d137e93c191e9f87fea3d778f2115f94afccf75955bc716d7094b6c3717c143" gracePeriod=30
Oct 30 22:31:10.665641 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:10.665606    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-4cm9t"]
Oct 30 22:31:10.672146 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:10.672109    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-4cm9t"
Oct 30 22:31:10.686126 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:10.686095    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-4cm9t"]
Oct 30 22:31:10.832594 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:10.832551    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c0da8b6-d70a-44bc-81cc-8e79df362b32-workload-socket\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-4cm9t\" (UID: \"1c0da8b6-d70a-44bc-81cc-8e79df362b32\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-4cm9t"
Oct 30 22:31:10.832737 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:10.832669    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c0da8b6-d70a-44bc-81cc-8e79df362b32-credential-socket\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-4cm9t\" (UID: \"1c0da8b6-d70a-44bc-81cc-8e79df362b32\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-4cm9t"
Oct 30 22:31:10.832737 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:10.832726    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c0da8b6-d70a-44bc-81cc-8e79df362b32-istiod-ca-cert\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-4cm9t\" (UID: \"1c0da8b6-d70a-44bc-81cc-8e79df362b32\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-4cm9t"
Oct 30 22:31:10.832825 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:10.832776    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c0da8b6-d70a-44bc-81cc-8e79df362b32-istio-envoy\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-4cm9t\" (UID: \"1c0da8b6-d70a-44bc-81cc-8e79df362b32\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-4cm9t"
Oct 30 22:31:10.832825 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:10.832810    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nh8tl\" (UniqueName: \"kubernetes.io/projected/1c0da8b6-d70a-44bc-81cc-8e79df362b32-kube-api-access-nh8tl\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-4cm9t\" (UID: \"1c0da8b6-d70a-44bc-81cc-8e79df362b32\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-4cm9t"
Oct 30 22:31:10.832890 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:10.832852    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c0da8b6-d70a-44bc-81cc-8e79df362b32-istio-token\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-4cm9t\" (UID: \"1c0da8b6-d70a-44bc-81cc-8e79df362b32\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-4cm9t"
Oct 30 22:31:10.832943 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:10.832912    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c0da8b6-d70a-44bc-81cc-8e79df362b32-workload-certs\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-4cm9t\" (UID: \"1c0da8b6-d70a-44bc-81cc-8e79df362b32\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-4cm9t"
Oct 30 22:31:10.832978 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:10.832964    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c0da8b6-d70a-44bc-81cc-8e79df362b32-istio-podinfo\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-4cm9t\" (UID: \"1c0da8b6-d70a-44bc-81cc-8e79df362b32\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-4cm9t"
Oct 30 22:31:10.833022 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:10.833012    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c0da8b6-d70a-44bc-81cc-8e79df362b32-istio-data\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-4cm9t\" (UID: \"1c0da8b6-d70a-44bc-81cc-8e79df362b32\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-4cm9t"
Oct 30 22:31:10.934286 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:10.934215    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c0da8b6-d70a-44bc-81cc-8e79df362b32-workload-socket\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-4cm9t\" (UID: \"1c0da8b6-d70a-44bc-81cc-8e79df362b32\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-4cm9t"
Oct 30 22:31:10.934286 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:10.934274    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c0da8b6-d70a-44bc-81cc-8e79df362b32-credential-socket\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-4cm9t\" (UID: \"1c0da8b6-d70a-44bc-81cc-8e79df362b32\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-4cm9t"
Oct 30 22:31:10.934495 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:10.934321    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c0da8b6-d70a-44bc-81cc-8e79df362b32-istiod-ca-cert\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-4cm9t\" (UID: \"1c0da8b6-d70a-44bc-81cc-8e79df362b32\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-4cm9t"
Oct 30 22:31:10.934495 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:10.934351    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c0da8b6-d70a-44bc-81cc-8e79df362b32-istio-envoy\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-4cm9t\" (UID: \"1c0da8b6-d70a-44bc-81cc-8e79df362b32\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-4cm9t"
Oct 30 22:31:10.934495 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:10.934390    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nh8tl\" (UniqueName: \"kubernetes.io/projected/1c0da8b6-d70a-44bc-81cc-8e79df362b32-kube-api-access-nh8tl\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-4cm9t\" (UID: \"1c0da8b6-d70a-44bc-81cc-8e79df362b32\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-4cm9t"
Oct 30 22:31:10.934495 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:10.934458    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c0da8b6-d70a-44bc-81cc-8e79df362b32-istio-token\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-4cm9t\" (UID: \"1c0da8b6-d70a-44bc-81cc-8e79df362b32\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-4cm9t"
Oct 30 22:31:10.934495 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:10.934485    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c0da8b6-d70a-44bc-81cc-8e79df362b32-workload-certs\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-4cm9t\" (UID: \"1c0da8b6-d70a-44bc-81cc-8e79df362b32\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-4cm9t"
Oct 30 22:31:10.934748 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:10.934624    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c0da8b6-d70a-44bc-81cc-8e79df362b32-istio-podinfo\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-4cm9t\" (UID: \"1c0da8b6-d70a-44bc-81cc-8e79df362b32\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-4cm9t"
Oct 30 22:31:10.934748 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:10.934685    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c0da8b6-d70a-44bc-81cc-8e79df362b32-workload-socket\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-4cm9t\" (UID: \"1c0da8b6-d70a-44bc-81cc-8e79df362b32\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-4cm9t"
Oct 30 22:31:10.934748 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:10.934696    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c0da8b6-d70a-44bc-81cc-8e79df362b32-istio-data\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-4cm9t\" (UID: \"1c0da8b6-d70a-44bc-81cc-8e79df362b32\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-4cm9t"
Oct 30 22:31:10.934900 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:10.934832    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c0da8b6-d70a-44bc-81cc-8e79df362b32-credential-socket\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-4cm9t\" (UID: \"1c0da8b6-d70a-44bc-81cc-8e79df362b32\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-4cm9t"
Oct 30 22:31:10.935115 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:10.935091    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c0da8b6-d70a-44bc-81cc-8e79df362b32-istio-data\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-4cm9t\" (UID: \"1c0da8b6-d70a-44bc-81cc-8e79df362b32\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-4cm9t"
Oct 30 22:31:10.935220 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:10.935135    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c0da8b6-d70a-44bc-81cc-8e79df362b32-istiod-ca-cert\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-4cm9t\" (UID: \"1c0da8b6-d70a-44bc-81cc-8e79df362b32\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-4cm9t"
Oct 30 22:31:10.935309 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:10.935287    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c0da8b6-d70a-44bc-81cc-8e79df362b32-workload-certs\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-4cm9t\" (UID: \"1c0da8b6-d70a-44bc-81cc-8e79df362b32\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-4cm9t"
Oct 30 22:31:10.936779 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:10.936754    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c0da8b6-d70a-44bc-81cc-8e79df362b32-istio-envoy\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-4cm9t\" (UID: \"1c0da8b6-d70a-44bc-81cc-8e79df362b32\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-4cm9t"
Oct 30 22:31:10.937244 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:10.937224    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c0da8b6-d70a-44bc-81cc-8e79df362b32-istio-podinfo\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-4cm9t\" (UID: \"1c0da8b6-d70a-44bc-81cc-8e79df362b32\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-4cm9t"
Oct 30 22:31:10.952278 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:10.952256    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c0da8b6-d70a-44bc-81cc-8e79df362b32-istio-token\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-4cm9t\" (UID: \"1c0da8b6-d70a-44bc-81cc-8e79df362b32\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-4cm9t"
Oct 30 22:31:10.952443 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:10.952425    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nh8tl\" (UniqueName: \"kubernetes.io/projected/1c0da8b6-d70a-44bc-81cc-8e79df362b32-kube-api-access-nh8tl\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-4cm9t\" (UID: \"1c0da8b6-d70a-44bc-81cc-8e79df362b32\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-4cm9t"
Oct 30 22:31:10.994434 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:10.994415    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-4cm9t"
Oct 30 22:31:11.309395 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:11.309369    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-4cm9t"]
Oct 30 22:31:11.312374 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:31:11.312343    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c0da8b6_d70a_44bc_81cc_8e79df362b32.slice/crio-186c3023acb3e3656b222886fb5f236668889e7bc187306e932d3a16dfd70f19 WatchSource:0}: Error finding container 186c3023acb3e3656b222886fb5f236668889e7bc187306e932d3a16dfd70f19: Status 404 returned error can't find the container with id 186c3023acb3e3656b222886fb5f236668889e7bc187306e932d3a16dfd70f19
Oct 30 22:31:11.315130 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:11.315092    4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:31:11.315198 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:11.315157    4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:31:11.315198 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:11.315187    4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:31:11.773766 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:11.773732    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-4cm9t" event={"ID":"1c0da8b6-d70a-44bc-81cc-8e79df362b32","Type":"ContainerStarted","Data":"a3cbda41180568f1eb03879a3c61938fc6a04eba33291b908909ff6dbef6b791"}
Oct 30 22:31:11.774279 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:11.773783    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-4cm9t" event={"ID":"1c0da8b6-d70a-44bc-81cc-8e79df362b32","Type":"ContainerStarted","Data":"186c3023acb3e3656b222886fb5f236668889e7bc187306e932d3a16dfd70f19"}
Oct 30 22:31:11.828062 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:11.827955    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-4cm9t" podStartSLOduration=1.827934704 podStartE2EDuration="1.827934704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:31:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:31:11.823779616 +0000 UTC m=+1455.749705859" watchObservedRunningTime="2026-10-30 22:31:11.827934704 +0000 UTC m=+1455.753860912"
Oct 30 22:31:11.995188 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:11.995068    4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-4cm9t"
Oct 30 22:31:12.996966 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:12.996933    4477 patch_prober.go:29] interesting pod/e2e-api-a-370ec2-openshift-default-6fccc4f697-4cm9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Oct 30 22:31:12.997425 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:12.997001    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-4cm9t" podUID="1c0da8b6-d70a-44bc-81cc-8e79df362b32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Oct 30 22:31:13.995512 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:13.995474    4477 patch_prober.go:29] interesting pod/e2e-api-a-370ec2-openshift-default-6fccc4f697-4cm9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Oct 30 22:31:13.995687 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:13.995543    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-4cm9t" podUID="1c0da8b6-d70a-44bc-81cc-8e79df362b32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Oct 30 22:31:14.507516 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:14.507490    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-z7ggc"
Oct 30 22:31:14.618699 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:14.618615    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5779d15a-20b4-4ec0-ac5e-b8bdfa89096d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5779d15a-20b4-4ec0-ac5e-b8bdfa89096d-istio-data\") pod \"5779d15a-20b4-4ec0-ac5e-b8bdfa89096d\" (UID: \"5779d15a-20b4-4ec0-ac5e-b8bdfa89096d\") "
Oct 30 22:31:14.618862 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:14.618707    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5779d15a-20b4-4ec0-ac5e-b8bdfa89096d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5779d15a-20b4-4ec0-ac5e-b8bdfa89096d-workload-socket\") pod \"5779d15a-20b4-4ec0-ac5e-b8bdfa89096d\" (UID: \"5779d15a-20b4-4ec0-ac5e-b8bdfa89096d\") "
Oct 30 22:31:14.618862 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:14.618784    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5779d15a-20b4-4ec0-ac5e-b8bdfa89096d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5779d15a-20b4-4ec0-ac5e-b8bdfa89096d-istio-podinfo\") pod \"5779d15a-20b4-4ec0-ac5e-b8bdfa89096d\" (UID: \"5779d15a-20b4-4ec0-ac5e-b8bdfa89096d\") "
Oct 30 22:31:14.618970 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:14.618880    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5779d15a-20b4-4ec0-ac5e-b8bdfa89096d-istio-data" pod "5779d15a-20b4-4ec0-ac5e-b8bdfa89096d" (UID: "5779d15a-20b4-4ec0-ac5e-b8bdfa89096d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:14.618970 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:14.618918    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5779d15a-20b4-4ec0-ac5e-b8bdfa89096d-kube-api-access-l2cg9\" (UniqueName: \"kubernetes.io/projected/5779d15a-20b4-4ec0-ac5e-b8bdfa89096d-kube-api-access-l2cg9\") pod \"5779d15a-20b4-4ec0-ac5e-b8bdfa89096d\" (UID: \"5779d15a-20b4-4ec0-ac5e-b8bdfa89096d\") "
Oct 30 22:31:14.619105 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:14.618980    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5779d15a-20b4-4ec0-ac5e-b8bdfa89096d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5779d15a-20b4-4ec0-ac5e-b8bdfa89096d-istiod-ca-cert\") pod \"5779d15a-20b4-4ec0-ac5e-b8bdfa89096d\" (UID: \"5779d15a-20b4-4ec0-ac5e-b8bdfa89096d\") "
Oct 30 22:31:14.619105 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:14.619015    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5779d15a-20b4-4ec0-ac5e-b8bdfa89096d-workload-socket" pod "5779d15a-20b4-4ec0-ac5e-b8bdfa89096d" (UID: "5779d15a-20b4-4ec0-ac5e-b8bdfa89096d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:14.619105 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:14.619032    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5779d15a-20b4-4ec0-ac5e-b8bdfa89096d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5779d15a-20b4-4ec0-ac5e-b8bdfa89096d-istio-envoy\") pod \"5779d15a-20b4-4ec0-ac5e-b8bdfa89096d\" (UID: \"5779d15a-20b4-4ec0-ac5e-b8bdfa89096d\") "
Oct 30 22:31:14.619252 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:14.619131    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5779d15a-20b4-4ec0-ac5e-b8bdfa89096d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5779d15a-20b4-4ec0-ac5e-b8bdfa89096d-workload-certs\") pod \"5779d15a-20b4-4ec0-ac5e-b8bdfa89096d\" (UID: \"5779d15a-20b4-4ec0-ac5e-b8bdfa89096d\") "
Oct 30 22:31:14.619252 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:14.619166    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5779d15a-20b4-4ec0-ac5e-b8bdfa89096d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5779d15a-20b4-4ec0-ac5e-b8bdfa89096d-credential-socket\") pod \"5779d15a-20b4-4ec0-ac5e-b8bdfa89096d\" (UID: \"5779d15a-20b4-4ec0-ac5e-b8bdfa89096d\") "
Oct 30 22:31:14.619354 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:14.619304    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5779d15a-20b4-4ec0-ac5e-b8bdfa89096d-istio-token\" (UniqueName: \"kubernetes.io/projected/5779d15a-20b4-4ec0-ac5e-b8bdfa89096d-istio-token\") pod \"5779d15a-20b4-4ec0-ac5e-b8bdfa89096d\" (UID: \"5779d15a-20b4-4ec0-ac5e-b8bdfa89096d\") "
Oct 30 22:31:14.620215 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:14.620188    4477 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5779d15a-20b4-4ec0-ac5e-b8bdfa89096d-istio-data\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:31:14.620354 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:14.620219    4477 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5779d15a-20b4-4ec0-ac5e-b8bdfa89096d-workload-socket\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:31:14.621066 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:14.621007    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5779d15a-20b4-4ec0-ac5e-b8bdfa89096d-istiod-ca-cert" pod "5779d15a-20b4-4ec0-ac5e-b8bdfa89096d" (UID: "5779d15a-20b4-4ec0-ac5e-b8bdfa89096d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:31:14.622158 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:14.622133    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5779d15a-20b4-4ec0-ac5e-b8bdfa89096d-credential-socket" pod "5779d15a-20b4-4ec0-ac5e-b8bdfa89096d" (UID: "5779d15a-20b4-4ec0-ac5e-b8bdfa89096d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:14.622585 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:14.622564    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5779d15a-20b4-4ec0-ac5e-b8bdfa89096d-workload-certs" pod "5779d15a-20b4-4ec0-ac5e-b8bdfa89096d" (UID: "5779d15a-20b4-4ec0-ac5e-b8bdfa89096d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:14.622800 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:14.622773    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5779d15a-20b4-4ec0-ac5e-b8bdfa89096d-istio-podinfo" pod "5779d15a-20b4-4ec0-ac5e-b8bdfa89096d" (UID: "5779d15a-20b4-4ec0-ac5e-b8bdfa89096d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:31:14.623345 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:14.623298    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5779d15a-20b4-4ec0-ac5e-b8bdfa89096d-istio-token" pod "5779d15a-20b4-4ec0-ac5e-b8bdfa89096d" (UID: "5779d15a-20b4-4ec0-ac5e-b8bdfa89096d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:31:14.623436 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:14.623345    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5779d15a-20b4-4ec0-ac5e-b8bdfa89096d-kube-api-access-l2cg9" pod "5779d15a-20b4-4ec0-ac5e-b8bdfa89096d" (UID: "5779d15a-20b4-4ec0-ac5e-b8bdfa89096d"). InnerVolumeSpecName "kube-api-access-l2cg9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:31:14.623436 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:14.623365    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5779d15a-20b4-4ec0-ac5e-b8bdfa89096d-istio-envoy" pod "5779d15a-20b4-4ec0-ac5e-b8bdfa89096d" (UID: "5779d15a-20b4-4ec0-ac5e-b8bdfa89096d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:14.721867 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:14.721833    4477 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5779d15a-20b4-4ec0-ac5e-b8bdfa89096d-istio-podinfo\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:31:14.721867 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:14.721863    4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2cg9\" (UniqueName: \"kubernetes.io/projected/5779d15a-20b4-4ec0-ac5e-b8bdfa89096d-kube-api-access-l2cg9\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:31:14.721867 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:14.721872    4477 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5779d15a-20b4-4ec0-ac5e-b8bdfa89096d-istiod-ca-cert\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:31:14.722126 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:14.721881    4477 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5779d15a-20b4-4ec0-ac5e-b8bdfa89096d-istio-envoy\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:31:14.722126 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:14.721890    4477 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5779d15a-20b4-4ec0-ac5e-b8bdfa89096d-workload-certs\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:31:14.722126 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:14.721897    4477 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5779d15a-20b4-4ec0-ac5e-b8bdfa89096d-credential-socket\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:31:14.722126 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:14.721907    4477 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5779d15a-20b4-4ec0-ac5e-b8bdfa89096d-istio-token\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:31:14.751086 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:14.751031    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-30edd7-5ff8d79b9d-2lqtr"
Oct 30 22:31:14.817133 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:14.817069    4477 generic.go:356] "Generic (PLEG): container finished" podID="5779d15a-20b4-4ec0-ac5e-b8bdfa89096d" containerID="2444cbe557ac681373e56ac69c36d693830c7a572f5cd296ec1b7c95dbaebe32" exitCode=0
Oct 30 22:31:14.817312 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:14.817162    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-z7ggc" event={"ID":"5779d15a-20b4-4ec0-ac5e-b8bdfa89096d","Type":"ContainerDied","Data":"2444cbe557ac681373e56ac69c36d693830c7a572f5cd296ec1b7c95dbaebe32"}
Oct 30 22:31:14.817312 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:14.817215    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-z7ggc" event={"ID":"5779d15a-20b4-4ec0-ac5e-b8bdfa89096d","Type":"ContainerDied","Data":"22cd51e1b69325ca20e47ad6a2d590e33e9d46ef006e0cd1dde5ad480bbc15ba"}
Oct 30 22:31:14.817312 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:14.817245    4477 scope.go:122] "RemoveContainer" containerID="2444cbe557ac681373e56ac69c36d693830c7a572f5cd296ec1b7c95dbaebe32"
Oct 30 22:31:14.817487 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:14.817416    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-z7ggc"
Oct 30 22:31:14.828307 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:14.828281    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bf9c2f-cfd8bcd86-pdn6v"]
Oct 30 22:31:14.829307 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:14.829275    4477 cpu_manager.go:414] "containerMap: removing container" podUID="5779d15a-20b4-4ec0-ac5e-b8bdfa89096d" containerName="istio-proxy"
Oct 30 22:31:14.829307 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:14.829300    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5779d15a-20b4-4ec0-ac5e-b8bdfa89096d" containerName="istio-proxy"
Oct 30 22:31:14.829518 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:14.829500    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="5779d15a-20b4-4ec0-ac5e-b8bdfa89096d" containerName="istio-proxy"
Oct 30 22:31:14.835957 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:14.833988    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bf9c2f-cfd8bcd86-pdn6v"
Oct 30 22:31:14.848906 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:14.848884    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-bf9c2f-serving-cert\""
Oct 30 22:31:14.850441 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:14.850420    4477 scope.go:122] "RemoveContainer" containerID="2444cbe557ac681373e56ac69c36d693830c7a572f5cd296ec1b7c95dbaebe32"
Oct 30 22:31:14.852890 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:31:14.852865    4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2444cbe557ac681373e56ac69c36d693830c7a572f5cd296ec1b7c95dbaebe32\": container with ID starting with 2444cbe557ac681373e56ac69c36d693830c7a572f5cd296ec1b7c95dbaebe32 not found: ID does not exist" containerID="2444cbe557ac681373e56ac69c36d693830c7a572f5cd296ec1b7c95dbaebe32"
Oct 30 22:31:14.853006 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:14.852901    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2444cbe557ac681373e56ac69c36d693830c7a572f5cd296ec1b7c95dbaebe32"} err="failed to get container status \"2444cbe557ac681373e56ac69c36d693830c7a572f5cd296ec1b7c95dbaebe32\": rpc error: code = NotFound desc = could not find container \"2444cbe557ac681373e56ac69c36d693830c7a572f5cd296ec1b7c95dbaebe32\": container with ID starting with 2444cbe557ac681373e56ac69c36d693830c7a572f5cd296ec1b7c95dbaebe32 not found: ID does not exist"
Oct 30 22:31:14.863302 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:14.863274    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bf9c2f-cfd8bcd86-pdn6v"]
Oct 30 22:31:14.939935 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:14.939174    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-z7ggc"]
Oct 30 22:31:14.946421 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:14.946393    4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-z7ggc"]
Oct 30 22:31:14.995406 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:14.995368    4477 patch_prober.go:29] interesting pod/e2e-api-a-370ec2-openshift-default-6fccc4f697-4cm9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Oct 30 22:31:14.995599 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:14.995578    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-4cm9t" podUID="1c0da8b6-d70a-44bc-81cc-8e79df362b32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Oct 30 22:31:15.027194 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:15.027033    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b450d906-242d-4c3e-97cd-1e805041f85a-openshift-service-ca\") pod \"maas-api-e2e-api-b-bf9c2f-cfd8bcd86-pdn6v\" (UID: \"b450d906-242d-4c3e-97cd-1e805041f85a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bf9c2f-cfd8bcd86-pdn6v"
Oct 30 22:31:15.027194 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:15.027142    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b450d906-242d-4c3e-97cd-1e805041f85a-maas-api-tls\") pod \"maas-api-e2e-api-b-bf9c2f-cfd8bcd86-pdn6v\" (UID: \"b450d906-242d-4c3e-97cd-1e805041f85a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bf9c2f-cfd8bcd86-pdn6v"
Oct 30 22:31:15.027432 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:15.027317    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b450d906-242d-4c3e-97cd-1e805041f85a-metrics-certs\") pod \"maas-api-e2e-api-b-bf9c2f-cfd8bcd86-pdn6v\" (UID: \"b450d906-242d-4c3e-97cd-1e805041f85a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bf9c2f-cfd8bcd86-pdn6v"
Oct 30 22:31:15.027432 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:15.027368    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrsgb\" (UniqueName: \"kubernetes.io/projected/b450d906-242d-4c3e-97cd-1e805041f85a-kube-api-access-hrsgb\") pod \"maas-api-e2e-api-b-bf9c2f-cfd8bcd86-pdn6v\" (UID: \"b450d906-242d-4c3e-97cd-1e805041f85a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bf9c2f-cfd8bcd86-pdn6v"
Oct 30 22:31:15.124716 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:15.124688    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-sqxvr"
Oct 30 22:31:15.128770 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:15.128600    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b450d906-242d-4c3e-97cd-1e805041f85a-openshift-service-ca\") pod \"maas-api-e2e-api-b-bf9c2f-cfd8bcd86-pdn6v\" (UID: \"b450d906-242d-4c3e-97cd-1e805041f85a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bf9c2f-cfd8bcd86-pdn6v"
Oct 30 22:31:15.128770 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:15.128697    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b450d906-242d-4c3e-97cd-1e805041f85a-maas-api-tls\") pod \"maas-api-e2e-api-b-bf9c2f-cfd8bcd86-pdn6v\" (UID: \"b450d906-242d-4c3e-97cd-1e805041f85a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bf9c2f-cfd8bcd86-pdn6v"
Oct 30 22:31:15.129169 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:15.129150    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b450d906-242d-4c3e-97cd-1e805041f85a-metrics-certs\") pod \"maas-api-e2e-api-b-bf9c2f-cfd8bcd86-pdn6v\" (UID: \"b450d906-242d-4c3e-97cd-1e805041f85a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bf9c2f-cfd8bcd86-pdn6v"
Oct 30 22:31:15.129277 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:15.129210    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrsgb\" (UniqueName: \"kubernetes.io/projected/b450d906-242d-4c3e-97cd-1e805041f85a-kube-api-access-hrsgb\") pod \"maas-api-e2e-api-b-bf9c2f-cfd8bcd86-pdn6v\" (UID: \"b450d906-242d-4c3e-97cd-1e805041f85a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bf9c2f-cfd8bcd86-pdn6v"
Oct 30 22:31:15.129428 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:15.129405    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b450d906-242d-4c3e-97cd-1e805041f85a-openshift-service-ca\") pod \"maas-api-e2e-api-b-bf9c2f-cfd8bcd86-pdn6v\" (UID: \"b450d906-242d-4c3e-97cd-1e805041f85a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bf9c2f-cfd8bcd86-pdn6v"
Oct 30 22:31:15.132259 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:15.132233    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b450d906-242d-4c3e-97cd-1e805041f85a-maas-api-tls\") pod \"maas-api-e2e-api-b-bf9c2f-cfd8bcd86-pdn6v\" (UID: \"b450d906-242d-4c3e-97cd-1e805041f85a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bf9c2f-cfd8bcd86-pdn6v"
Oct 30 22:31:15.132349 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:15.132241    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b450d906-242d-4c3e-97cd-1e805041f85a-metrics-certs\") pod \"maas-api-e2e-api-b-bf9c2f-cfd8bcd86-pdn6v\" (UID: \"b450d906-242d-4c3e-97cd-1e805041f85a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bf9c2f-cfd8bcd86-pdn6v"
Oct 30 22:31:15.147498 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:15.147470    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrsgb\" (UniqueName: \"kubernetes.io/projected/b450d906-242d-4c3e-97cd-1e805041f85a-kube-api-access-hrsgb\") pod \"maas-api-e2e-api-b-bf9c2f-cfd8bcd86-pdn6v\" (UID: \"b450d906-242d-4c3e-97cd-1e805041f85a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bf9c2f-cfd8bcd86-pdn6v"
Oct 30 22:31:15.160770 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:15.160750    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bf9c2f-cfd8bcd86-pdn6v"
Oct 30 22:31:15.238376 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:15.238341    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/909907e9-319a-48dc-90d6-735cee614217-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/909907e9-319a-48dc-90d6-735cee614217-istio-podinfo\") pod \"909907e9-319a-48dc-90d6-735cee614217\" (UID: \"909907e9-319a-48dc-90d6-735cee614217\") "
Oct 30 22:31:15.238542 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:15.238477    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/909907e9-319a-48dc-90d6-735cee614217-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/909907e9-319a-48dc-90d6-735cee614217-workload-socket\") pod \"909907e9-319a-48dc-90d6-735cee614217\" (UID: \"909907e9-319a-48dc-90d6-735cee614217\") "
Oct 30 22:31:15.238542 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:15.238529    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/909907e9-319a-48dc-90d6-735cee614217-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/909907e9-319a-48dc-90d6-735cee614217-istio-envoy\") pod \"909907e9-319a-48dc-90d6-735cee614217\" (UID: \"909907e9-319a-48dc-90d6-735cee614217\") "
Oct 30 22:31:15.238659 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:15.238567    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/909907e9-319a-48dc-90d6-735cee614217-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/909907e9-319a-48dc-90d6-735cee614217-credential-socket\") pod \"909907e9-319a-48dc-90d6-735cee614217\" (UID: \"909907e9-319a-48dc-90d6-735cee614217\") "
Oct 30 22:31:15.238659 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:15.238622    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/909907e9-319a-48dc-90d6-735cee614217-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/909907e9-319a-48dc-90d6-735cee614217-workload-certs\") pod \"909907e9-319a-48dc-90d6-735cee614217\" (UID: \"909907e9-319a-48dc-90d6-735cee614217\") "
Oct 30 22:31:15.238762 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:15.238677    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/909907e9-319a-48dc-90d6-735cee614217-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/909907e9-319a-48dc-90d6-735cee614217-istiod-ca-cert\") pod \"909907e9-319a-48dc-90d6-735cee614217\" (UID: \"909907e9-319a-48dc-90d6-735cee614217\") "
Oct 30 22:31:15.238762 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:15.238711    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/909907e9-319a-48dc-90d6-735cee614217-kube-api-access-vvrt2\" (UniqueName: \"kubernetes.io/projected/909907e9-319a-48dc-90d6-735cee614217-kube-api-access-vvrt2\") pod \"909907e9-319a-48dc-90d6-735cee614217\" (UID: \"909907e9-319a-48dc-90d6-735cee614217\") "
Oct 30 22:31:15.238762 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:15.238755    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/909907e9-319a-48dc-90d6-735cee614217-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/909907e9-319a-48dc-90d6-735cee614217-istio-data\") pod \"909907e9-319a-48dc-90d6-735cee614217\" (UID: \"909907e9-319a-48dc-90d6-735cee614217\") "
Oct 30 22:31:15.238915 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:15.238815    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/909907e9-319a-48dc-90d6-735cee614217-istio-token\" (UniqueName: \"kubernetes.io/projected/909907e9-319a-48dc-90d6-735cee614217-istio-token\") pod \"909907e9-319a-48dc-90d6-735cee614217\" (UID: \"909907e9-319a-48dc-90d6-735cee614217\") "
Oct 30 22:31:15.241353 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:15.241320    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/909907e9-319a-48dc-90d6-735cee614217-workload-certs" pod "909907e9-319a-48dc-90d6-735cee614217" (UID: "909907e9-319a-48dc-90d6-735cee614217"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:15.242582 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:15.242213    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/909907e9-319a-48dc-90d6-735cee614217-credential-socket" pod "909907e9-319a-48dc-90d6-735cee614217" (UID: "909907e9-319a-48dc-90d6-735cee614217"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:15.242582 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:15.242522    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/909907e9-319a-48dc-90d6-735cee614217-workload-socket" pod "909907e9-319a-48dc-90d6-735cee614217" (UID: "909907e9-319a-48dc-90d6-735cee614217"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:15.243367 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:15.243091    4477 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/909907e9-319a-48dc-90d6-735cee614217-workload-socket\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:31:15.243367 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:15.243114    4477 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/909907e9-319a-48dc-90d6-735cee614217-credential-socket\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:31:15.243367 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:15.243130    4477 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/909907e9-319a-48dc-90d6-735cee614217-workload-certs\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:31:15.243367 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:15.243205    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/909907e9-319a-48dc-90d6-735cee614217-istio-data" pod "909907e9-319a-48dc-90d6-735cee614217" (UID: "909907e9-319a-48dc-90d6-735cee614217"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:15.244233 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:15.244206    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/909907e9-319a-48dc-90d6-735cee614217-kube-api-access-vvrt2" pod "909907e9-319a-48dc-90d6-735cee614217" (UID: "909907e9-319a-48dc-90d6-735cee614217"). InnerVolumeSpecName "kube-api-access-vvrt2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:31:15.244354 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:15.244335    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/909907e9-319a-48dc-90d6-735cee614217-istiod-ca-cert" pod "909907e9-319a-48dc-90d6-735cee614217" (UID: "909907e9-319a-48dc-90d6-735cee614217"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:31:15.245550 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:15.244907    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/909907e9-319a-48dc-90d6-735cee614217-istio-podinfo" pod "909907e9-319a-48dc-90d6-735cee614217" (UID: "909907e9-319a-48dc-90d6-735cee614217"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:31:15.245550 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:15.245437    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/909907e9-319a-48dc-90d6-735cee614217-istio-envoy" pod "909907e9-319a-48dc-90d6-735cee614217" (UID: "909907e9-319a-48dc-90d6-735cee614217"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:15.247258 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:15.245956    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/909907e9-319a-48dc-90d6-735cee614217-istio-token" pod "909907e9-319a-48dc-90d6-735cee614217" (UID: "909907e9-319a-48dc-90d6-735cee614217"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:31:15.345955 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:15.345925    4477 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/909907e9-319a-48dc-90d6-735cee614217-istio-envoy\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:31:15.345955 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:15.345955    4477 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/909907e9-319a-48dc-90d6-735cee614217-istiod-ca-cert\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:31:15.346189 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:15.345964    4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvrt2\" (UniqueName: \"kubernetes.io/projected/909907e9-319a-48dc-90d6-735cee614217-kube-api-access-vvrt2\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:31:15.346189 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:15.345972    4477 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/909907e9-319a-48dc-90d6-735cee614217-istio-data\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:31:15.346189 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:15.345980    4477 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/909907e9-319a-48dc-90d6-735cee614217-istio-token\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:31:15.346189 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:15.345988    4477 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/909907e9-319a-48dc-90d6-735cee614217-istio-podinfo\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:31:15.563906 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:15.563824    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bf9c2f-cfd8bcd86-pdn6v"]
Oct 30 22:31:15.567682 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:31:15.567646    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb450d906_242d_4c3e_97cd_1e805041f85a.slice/crio-a541c7e442d3965129509ab110ad25b733af8ea1098f702def9511893b2e9b5d WatchSource:0}: Error finding container a541c7e442d3965129509ab110ad25b733af8ea1098f702def9511893b2e9b5d: Status 404 returned error can't find the container with id a541c7e442d3965129509ab110ad25b733af8ea1098f702def9511893b2e9b5d
Oct 30 22:31:15.824655 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:15.824569    4477 generic.go:356] "Generic (PLEG): container finished" podID="909907e9-319a-48dc-90d6-735cee614217" containerID="5d137e93c191e9f87fea3d778f2115f94afccf75955bc716d7094b6c3717c143" exitCode=0
Oct 30 22:31:15.824655 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:15.824622    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-sqxvr"
Oct 30 22:31:15.824868 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:15.824652    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-sqxvr" event={"ID":"909907e9-319a-48dc-90d6-735cee614217","Type":"ContainerDied","Data":"5d137e93c191e9f87fea3d778f2115f94afccf75955bc716d7094b6c3717c143"}
Oct 30 22:31:15.824868 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:15.824708    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-sqxvr" event={"ID":"909907e9-319a-48dc-90d6-735cee614217","Type":"ContainerDied","Data":"68754249f74b3d328fb200ef5059eb3e8f621f8824d4cf09e5843a585e84c5db"}
Oct 30 22:31:15.824868 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:15.824737    4477 scope.go:122] "RemoveContainer" containerID="5d137e93c191e9f87fea3d778f2115f94afccf75955bc716d7094b6c3717c143"
Oct 30 22:31:15.826109 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:15.826085    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bf9c2f-cfd8bcd86-pdn6v" event={"ID":"b450d906-242d-4c3e-97cd-1e805041f85a","Type":"ContainerStarted","Data":"a541c7e442d3965129509ab110ad25b733af8ea1098f702def9511893b2e9b5d"}
Oct 30 22:31:15.835216 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:15.835198    4477 scope.go:122] "RemoveContainer" containerID="5d137e93c191e9f87fea3d778f2115f94afccf75955bc716d7094b6c3717c143"
Oct 30 22:31:15.835493 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:31:15.835475    4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d137e93c191e9f87fea3d778f2115f94afccf75955bc716d7094b6c3717c143\": container with ID starting with 5d137e93c191e9f87fea3d778f2115f94afccf75955bc716d7094b6c3717c143 not found: ID does not exist" containerID="5d137e93c191e9f87fea3d778f2115f94afccf75955bc716d7094b6c3717c143"
Oct 30 22:31:15.835537 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:15.835501    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d137e93c191e9f87fea3d778f2115f94afccf75955bc716d7094b6c3717c143"} err="failed to get container status \"5d137e93c191e9f87fea3d778f2115f94afccf75955bc716d7094b6c3717c143\": rpc error: code = NotFound desc = could not find container \"5d137e93c191e9f87fea3d778f2115f94afccf75955bc716d7094b6c3717c143\": container with ID starting with 5d137e93c191e9f87fea3d778f2115f94afccf75955bc716d7094b6c3717c143 not found: ID does not exist"
Oct 30 22:31:15.901242 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:15.887813    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-sqxvr"]
Oct 30 22:31:15.906302 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:15.906270    4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-sqxvr"]
Oct 30 22:31:16.011065 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:15.999153    4477 patch_prober.go:29] interesting pod/e2e-api-a-370ec2-openshift-default-6fccc4f697-4cm9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Oct 30 22:31:16.011065 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:15.999209    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-4cm9t" podUID="1c0da8b6-d70a-44bc-81cc-8e79df362b32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Oct 30 22:31:16.229098 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:16.228627    4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5779d15a-20b4-4ec0-ac5e-b8bdfa89096d" path="/var/lib/kubelet/pods/5779d15a-20b4-4ec0-ac5e-b8bdfa89096d/volumes"
Oct 30 22:31:16.229263 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:16.229167    4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="909907e9-319a-48dc-90d6-735cee614217" path="/var/lib/kubelet/pods/909907e9-319a-48dc-90d6-735cee614217/volumes"
Oct 30 22:31:16.841528 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:16.841481    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bf9c2f-cfd8bcd86-pdn6v" event={"ID":"b450d906-242d-4c3e-97cd-1e805041f85a","Type":"ContainerStarted","Data":"f789827d6171ac05f2a5f959b789997ebe0470542314912708fa9470abca99bc"}
Oct 30 22:31:16.842121 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:16.841581    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bf9c2f-cfd8bcd86-pdn6v"
Oct 30 22:31:16.893064 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:16.892748    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bf9c2f-cfd8bcd86-pdn6v" podStartSLOduration=2.5585524619999997 podStartE2EDuration="2.892731604s" totalImagesPullingTime="334.179142ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:31:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:31:16.892412916 +0000 UTC m=+1460.818339139" watchObservedRunningTime="2026-10-30 22:31:16.892731604 +0000 UTC m=+1460.818657812"
Oct 30 22:31:16.995828 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:16.995648    4477 patch_prober.go:29] interesting pod/e2e-api-a-370ec2-openshift-default-6fccc4f697-4cm9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused" start-of-body=
Oct 30 22:31:16.995828 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:16.995704    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-4cm9t" podUID="1c0da8b6-d70a-44bc-81cc-8e79df362b32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": dial tcp 10.134.0.102:15021: connect: connection refused"
Oct 30 22:31:18.000378 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:18.000347    4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-4cm9t"
Oct 30 22:31:18.000784 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:18.000760    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-4cm9t"
Oct 30 22:31:18.001376 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:18.001358    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-4cm9t"
Oct 30 22:31:22.857449 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:22.857418    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bf9c2f-cfd8bcd86-pdn6v"
Oct 30 22:31:25.693959 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:25.693925    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-2bzhm"]
Oct 30 22:31:25.694663 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:25.694646    4477 cpu_manager.go:414] "containerMap: removing container" podUID="909907e9-319a-48dc-90d6-735cee614217" containerName="istio-proxy"
Oct 30 22:31:25.694711 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:25.694666    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="909907e9-319a-48dc-90d6-735cee614217" containerName="istio-proxy"
Oct 30 22:31:25.694804 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:25.694793    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="909907e9-319a-48dc-90d6-735cee614217" containerName="istio-proxy"
Oct 30 22:31:25.698055 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:25.698024    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-2bzhm"
Oct 30 22:31:25.828798 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:25.828763    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-2bzhm"]
Oct 30 22:31:25.841111 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:25.830093    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92-credential-socket\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-2bzhm\" (UID: \"fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-2bzhm"
Oct 30 22:31:25.841464 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:25.841435    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92-workload-socket\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-2bzhm\" (UID: \"fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-2bzhm"
Oct 30 22:31:25.841617 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:25.841598    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92-istio-podinfo\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-2bzhm\" (UID: \"fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-2bzhm"
Oct 30 22:31:25.842308 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:25.842283    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92-istio-token\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-2bzhm\" (UID: \"fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-2bzhm"
Oct 30 22:31:25.842681 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:25.842660    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92-istiod-ca-cert\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-2bzhm\" (UID: \"fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-2bzhm"
Oct 30 22:31:25.842878 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:25.842861    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92-istio-envoy\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-2bzhm\" (UID: \"fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-2bzhm"
Oct 30 22:31:25.843034 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:25.843017    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nc85x\" (UniqueName: \"kubernetes.io/projected/fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92-kube-api-access-nc85x\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-2bzhm\" (UID: \"fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-2bzhm"
Oct 30 22:31:25.843258 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:25.843240    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92-workload-certs\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-2bzhm\" (UID: \"fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-2bzhm"
Oct 30 22:31:25.843463 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:25.843439    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92-istio-data\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-2bzhm\" (UID: \"fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-2bzhm"
Oct 30 22:31:25.944715 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:25.944611    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92-istio-data\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-2bzhm\" (UID: \"fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-2bzhm"
Oct 30 22:31:25.944900 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:25.944727    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92-credential-socket\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-2bzhm\" (UID: \"fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-2bzhm"
Oct 30 22:31:25.944900 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:25.944789    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92-workload-socket\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-2bzhm\" (UID: \"fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-2bzhm"
Oct 30 22:31:25.944900 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:25.944821    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92-istio-podinfo\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-2bzhm\" (UID: \"fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-2bzhm"
Oct 30 22:31:25.945174 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:25.944941    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92-istio-token\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-2bzhm\" (UID: \"fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-2bzhm"
Oct 30 22:31:25.945174 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:25.944998    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92-istiod-ca-cert\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-2bzhm\" (UID: \"fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-2bzhm"
Oct 30 22:31:25.945174 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:25.945031    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92-istio-envoy\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-2bzhm\" (UID: \"fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-2bzhm"
Oct 30 22:31:25.945174 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:25.945099    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92-istio-data\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-2bzhm\" (UID: \"fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-2bzhm"
Oct 30 22:31:25.945382 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:25.945182    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92-credential-socket\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-2bzhm\" (UID: \"fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-2bzhm"
Oct 30 22:31:25.945382 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:25.945204    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92-workload-socket\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-2bzhm\" (UID: \"fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-2bzhm"
Oct 30 22:31:25.945794 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:25.945767    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nc85x\" (UniqueName: \"kubernetes.io/projected/fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92-kube-api-access-nc85x\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-2bzhm\" (UID: \"fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-2bzhm"
Oct 30 22:31:25.945794 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:25.945785    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92-istiod-ca-cert\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-2bzhm\" (UID: \"fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-2bzhm"
Oct 30 22:31:25.946901 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:25.945869    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92-workload-certs\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-2bzhm\" (UID: \"fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-2bzhm"
Oct 30 22:31:25.946901 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:25.946260    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92-workload-certs\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-2bzhm\" (UID: \"fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-2bzhm"
Oct 30 22:31:25.947794 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:25.947764    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92-istio-envoy\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-2bzhm\" (UID: \"fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-2bzhm"
Oct 30 22:31:25.948335 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:25.948312    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92-istio-podinfo\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-2bzhm\" (UID: \"fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-2bzhm"
Oct 30 22:31:25.965537 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:25.965513    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nc85x\" (UniqueName: \"kubernetes.io/projected/fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92-kube-api-access-nc85x\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-2bzhm\" (UID: \"fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-2bzhm"
Oct 30 22:31:25.966378 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:25.966354    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92-istio-token\") pod \"e2e-api-a-370ec2-openshift-default-6fccc4f697-2bzhm\" (UID: \"fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92\") " pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-2bzhm"
Oct 30 22:31:26.042525 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:26.042493    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-2bzhm"
Oct 30 22:31:26.267767 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:31:26.267684    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 22:31:26.371598 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:26.360941    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-2bzhm"]
Oct 30 22:31:26.371598 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:31:26.363846    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe4a6d2a_5ccd_400d_9948_c7da6ffa4f92.slice/crio-a0f080387c73f40a1d4c1660da44ea045deb43379a835c0dff38ce992af67f69 WatchSource:0}: Error finding container a0f080387c73f40a1d4c1660da44ea045deb43379a835c0dff38ce992af67f69: Status 404 returned error can't find the container with id a0f080387c73f40a1d4c1660da44ea045deb43379a835c0dff38ce992af67f69
Oct 30 22:31:26.371598 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:26.367000    4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:31:26.371598 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:26.367080    4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:31:26.371598 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:26.367109    4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:31:26.924608 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:26.924569    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-2bzhm" event={"ID":"fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92","Type":"ContainerStarted","Data":"08dd29e2e4f609d04f560a5688d0ea2a2aaf0f43b6aac4dcd86b8129c0ef0c6f"}
Oct 30 22:31:26.925015 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:26.924623    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-2bzhm" event={"ID":"fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92","Type":"ContainerStarted","Data":"a0f080387c73f40a1d4c1660da44ea045deb43379a835c0dff38ce992af67f69"}
Oct 30 22:31:27.042733 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:27.042702    4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-2bzhm"
Oct 30 22:31:27.043989 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:27.043964    4477 patch_prober.go:29] interesting pod/e2e-api-a-370ec2-openshift-default-6fccc4f697-2bzhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body=
Oct 30 22:31:27.044095 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:27.044013    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-2bzhm" podUID="fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused"
Oct 30 22:31:28.109215 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:28.107904    4477 patch_prober.go:29] interesting pod/e2e-api-a-370ec2-openshift-default-6fccc4f697-2bzhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body=
Oct 30 22:31:28.109215 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:28.107969    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-2bzhm" podUID="fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused"
Oct 30 22:31:29.043678 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:29.043639    4477 patch_prober.go:29] interesting pod/e2e-api-a-370ec2-openshift-default-6fccc4f697-2bzhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body=
Oct 30 22:31:29.043875 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:29.043698    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-2bzhm" podUID="fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused"
Oct 30 22:31:29.956907 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:29.956839    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-2bzhm" podStartSLOduration=4.9568189799999995 podStartE2EDuration="4.95681898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:31:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:31:26.986425491 +0000 UTC m=+1470.912351717" watchObservedRunningTime="2026-10-30 22:31:29.95681898 +0000 UTC m=+1473.882745188"
Oct 30 22:31:29.958925 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:29.958895    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-7a1ee7-787c657595-2vdgl"]
Oct 30 22:31:29.964422 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:29.964404    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7a1ee7-787c657595-2vdgl"
Oct 30 22:31:29.986955 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:29.986928    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-7a1ee7-serving-cert\""
Oct 30 22:31:30.021121 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:30.018137    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-7a1ee7-787c657595-2vdgl"]
Oct 30 22:31:30.035562 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:30.035527    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ffd9f8a7-2785-4a15-8121-ff13661dadb1-maas-api-tls\") pod \"maas-api-e2e-api-a-7a1ee7-787c657595-2vdgl\" (UID: \"ffd9f8a7-2785-4a15-8121-ff13661dadb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7a1ee7-787c657595-2vdgl"
Oct 30 22:31:30.035713 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:30.035627    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w76k5\" (UniqueName: \"kubernetes.io/projected/ffd9f8a7-2785-4a15-8121-ff13661dadb1-kube-api-access-w76k5\") pod \"maas-api-e2e-api-a-7a1ee7-787c657595-2vdgl\" (UID: \"ffd9f8a7-2785-4a15-8121-ff13661dadb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7a1ee7-787c657595-2vdgl"
Oct 30 22:31:30.035797 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:30.035780    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffd9f8a7-2785-4a15-8121-ff13661dadb1-metrics-certs\") pod \"maas-api-e2e-api-a-7a1ee7-787c657595-2vdgl\" (UID: \"ffd9f8a7-2785-4a15-8121-ff13661dadb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7a1ee7-787c657595-2vdgl"
Oct 30 22:31:30.035869 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:30.035817    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffd9f8a7-2785-4a15-8121-ff13661dadb1-openshift-service-ca\") pod \"maas-api-e2e-api-a-7a1ee7-787c657595-2vdgl\" (UID: \"ffd9f8a7-2785-4a15-8121-ff13661dadb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7a1ee7-787c657595-2vdgl"
Oct 30 22:31:30.043496 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:30.043474    4477 patch_prober.go:29] interesting pod/e2e-api-a-370ec2-openshift-default-6fccc4f697-2bzhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body=
Oct 30 22:31:30.043614 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:30.043517    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-2bzhm" podUID="fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused"
Oct 30 22:31:30.136945 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:30.136914    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w76k5\" (UniqueName: \"kubernetes.io/projected/ffd9f8a7-2785-4a15-8121-ff13661dadb1-kube-api-access-w76k5\") pod \"maas-api-e2e-api-a-7a1ee7-787c657595-2vdgl\" (UID: \"ffd9f8a7-2785-4a15-8121-ff13661dadb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7a1ee7-787c657595-2vdgl"
Oct 30 22:31:30.137153 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:30.137107    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffd9f8a7-2785-4a15-8121-ff13661dadb1-metrics-certs\") pod \"maas-api-e2e-api-a-7a1ee7-787c657595-2vdgl\" (UID: \"ffd9f8a7-2785-4a15-8121-ff13661dadb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7a1ee7-787c657595-2vdgl"
Oct 30 22:31:30.137233 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:30.137158    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffd9f8a7-2785-4a15-8121-ff13661dadb1-openshift-service-ca\") pod \"maas-api-e2e-api-a-7a1ee7-787c657595-2vdgl\" (UID: \"ffd9f8a7-2785-4a15-8121-ff13661dadb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7a1ee7-787c657595-2vdgl"
Oct 30 22:31:30.137321 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:30.137306    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ffd9f8a7-2785-4a15-8121-ff13661dadb1-maas-api-tls\") pod \"maas-api-e2e-api-a-7a1ee7-787c657595-2vdgl\" (UID: \"ffd9f8a7-2785-4a15-8121-ff13661dadb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7a1ee7-787c657595-2vdgl"
Oct 30 22:31:30.137880 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:30.137855    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffd9f8a7-2785-4a15-8121-ff13661dadb1-openshift-service-ca\") pod \"maas-api-e2e-api-a-7a1ee7-787c657595-2vdgl\" (UID: \"ffd9f8a7-2785-4a15-8121-ff13661dadb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7a1ee7-787c657595-2vdgl"
Oct 30 22:31:30.139601 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:30.139574    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffd9f8a7-2785-4a15-8121-ff13661dadb1-metrics-certs\") pod \"maas-api-e2e-api-a-7a1ee7-787c657595-2vdgl\" (UID: \"ffd9f8a7-2785-4a15-8121-ff13661dadb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7a1ee7-787c657595-2vdgl"
Oct 30 22:31:30.139843 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:30.139824    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ffd9f8a7-2785-4a15-8121-ff13661dadb1-maas-api-tls\") pod \"maas-api-e2e-api-a-7a1ee7-787c657595-2vdgl\" (UID: \"ffd9f8a7-2785-4a15-8121-ff13661dadb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7a1ee7-787c657595-2vdgl"
Oct 30 22:31:30.154296 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:30.154273    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w76k5\" (UniqueName: \"kubernetes.io/projected/ffd9f8a7-2785-4a15-8121-ff13661dadb1-kube-api-access-w76k5\") pod \"maas-api-e2e-api-a-7a1ee7-787c657595-2vdgl\" (UID: \"ffd9f8a7-2785-4a15-8121-ff13661dadb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7a1ee7-787c657595-2vdgl"
Oct 30 22:31:30.330856 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:30.330771    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7a1ee7-787c657595-2vdgl"
Oct 30 22:31:30.696367 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:30.696339    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-7a1ee7-787c657595-2vdgl"]
Oct 30 22:31:30.698716 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:31:30.698684    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffd9f8a7_2785_4a15_8121_ff13661dadb1.slice/crio-4bb8688ce73ad33739c1c9ac4c33b01fafb070509121a60a920be92ae9bf95ff WatchSource:0}: Error finding container 4bb8688ce73ad33739c1c9ac4c33b01fafb070509121a60a920be92ae9bf95ff: Status 404 returned error can't find the container with id 4bb8688ce73ad33739c1c9ac4c33b01fafb070509121a60a920be92ae9bf95ff
Oct 30 22:31:31.044486 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:31.044305    4477 patch_prober.go:29] interesting pod/e2e-api-a-370ec2-openshift-default-6fccc4f697-2bzhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body=
Oct 30 22:31:31.044486 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:31.044363    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-2bzhm" podUID="fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused"
Oct 30 22:31:31.125800 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:31.125759    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7a1ee7-787c657595-2vdgl" event={"ID":"ffd9f8a7-2785-4a15-8121-ff13661dadb1","Type":"ContainerStarted","Data":"4bb8688ce73ad33739c1c9ac4c33b01fafb070509121a60a920be92ae9bf95ff"}
Oct 30 22:31:32.043673 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:32.043642    4477 patch_prober.go:29] interesting pod/e2e-api-a-370ec2-openshift-default-6fccc4f697-2bzhm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body=
Oct 30 22:31:32.043840 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:32.043694    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-2bzhm" podUID="fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused"
Oct 30 22:31:32.133208 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:32.133172    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7a1ee7-787c657595-2vdgl" event={"ID":"ffd9f8a7-2785-4a15-8121-ff13661dadb1","Type":"ContainerStarted","Data":"499fc8694791b5281e6e8a00fe488a9aff57f028e795aafa03e10f8c1322d598"}
Oct 30 22:31:32.133604 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:32.133398    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7a1ee7-787c657595-2vdgl"
Oct 30 22:31:32.199063 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:32.198129    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7a1ee7-787c657595-2vdgl" podStartSLOduration=2.675547324 podStartE2EDuration="3.198108949s" totalImagesPullingTime="522.561625ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:31:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:31:32.169246252 +0000 UTC m=+1476.095172482" watchObservedRunningTime="2026-10-30 22:31:32.198108949 +0000 UTC m=+1476.124035156"
Oct 30 22:31:33.124081 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:33.124009    4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-2bzhm"
Oct 30 22:31:33.124504 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:33.124477    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-2bzhm"
Oct 30 22:31:33.125455 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:33.125427    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-2bzhm"
Oct 30 22:31:38.146441 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:38.146412    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7a1ee7-787c657595-2vdgl"
Oct 30 22:31:39.217878 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:39.217809    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-3d38f6-74d6675f9d-sflzt"]
Oct 30 22:31:39.222597 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:39.222573    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3d38f6-74d6675f9d-sflzt"
Oct 30 22:31:39.257105 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:39.240770    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-3d38f6-serving-cert\""
Oct 30 22:31:39.346209 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:39.346168    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-3d38f6-74d6675f9d-sflzt"]
Oct 30 22:31:39.452920 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:39.452885    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6gdm\" (UniqueName: \"kubernetes.io/projected/46d350c3-9eff-403a-844d-f50edc815b45-kube-api-access-q6gdm\") pod \"maas-api-e2e-api-b-3d38f6-74d6675f9d-sflzt\" (UID: \"46d350c3-9eff-403a-844d-f50edc815b45\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3d38f6-74d6675f9d-sflzt"
Oct 30 22:31:39.453143 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:39.452947    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46d350c3-9eff-403a-844d-f50edc815b45-openshift-service-ca\") pod \"maas-api-e2e-api-b-3d38f6-74d6675f9d-sflzt\" (UID: \"46d350c3-9eff-403a-844d-f50edc815b45\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3d38f6-74d6675f9d-sflzt"
Oct 30 22:31:39.453442 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:39.453419    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46d350c3-9eff-403a-844d-f50edc815b45-metrics-certs\") pod \"maas-api-e2e-api-b-3d38f6-74d6675f9d-sflzt\" (UID: \"46d350c3-9eff-403a-844d-f50edc815b45\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3d38f6-74d6675f9d-sflzt"
Oct 30 22:31:39.453618 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:39.453599    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46d350c3-9eff-403a-844d-f50edc815b45-maas-api-tls\") pod \"maas-api-e2e-api-b-3d38f6-74d6675f9d-sflzt\" (UID: \"46d350c3-9eff-403a-844d-f50edc815b45\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3d38f6-74d6675f9d-sflzt"
Oct 30 22:31:39.554923 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:39.554836    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46d350c3-9eff-403a-844d-f50edc815b45-metrics-certs\") pod \"maas-api-e2e-api-b-3d38f6-74d6675f9d-sflzt\" (UID: \"46d350c3-9eff-403a-844d-f50edc815b45\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3d38f6-74d6675f9d-sflzt"
Oct 30 22:31:39.555103 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:39.554956    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46d350c3-9eff-403a-844d-f50edc815b45-maas-api-tls\") pod \"maas-api-e2e-api-b-3d38f6-74d6675f9d-sflzt\" (UID: \"46d350c3-9eff-403a-844d-f50edc815b45\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3d38f6-74d6675f9d-sflzt"
Oct 30 22:31:39.555103 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:39.555093    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6gdm\" (UniqueName: \"kubernetes.io/projected/46d350c3-9eff-403a-844d-f50edc815b45-kube-api-access-q6gdm\") pod \"maas-api-e2e-api-b-3d38f6-74d6675f9d-sflzt\" (UID: \"46d350c3-9eff-403a-844d-f50edc815b45\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3d38f6-74d6675f9d-sflzt"
Oct 30 22:31:39.555247 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:31:39.555123    4477 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-3d38f6-serving-cert: secret "maas-api-e2e-api-b-3d38f6-serving-cert" not found
Oct 30 22:31:39.555247 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:39.555142    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46d350c3-9eff-403a-844d-f50edc815b45-openshift-service-ca\") pod \"maas-api-e2e-api-b-3d38f6-74d6675f9d-sflzt\" (UID: \"46d350c3-9eff-403a-844d-f50edc815b45\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3d38f6-74d6675f9d-sflzt"
Oct 30 22:31:39.555247 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:31:39.555216    4477 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46d350c3-9eff-403a-844d-f50edc815b45-maas-api-tls podName:46d350c3-9eff-403a-844d-f50edc815b45 nodeName:}" failed. No retries permitted until 2026-10-30 22:31:40.055192152 +0000 UTC m=+1483.981118344 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/46d350c3-9eff-403a-844d-f50edc815b45-maas-api-tls") pod "maas-api-e2e-api-b-3d38f6-74d6675f9d-sflzt" (UID: "46d350c3-9eff-403a-844d-f50edc815b45") : secret "maas-api-e2e-api-b-3d38f6-serving-cert" not found
Oct 30 22:31:39.555891 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:39.555862    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46d350c3-9eff-403a-844d-f50edc815b45-openshift-service-ca\") pod \"maas-api-e2e-api-b-3d38f6-74d6675f9d-sflzt\" (UID: \"46d350c3-9eff-403a-844d-f50edc815b45\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3d38f6-74d6675f9d-sflzt"
Oct 30 22:31:39.557460 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:39.557435    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46d350c3-9eff-403a-844d-f50edc815b45-metrics-certs\") pod \"maas-api-e2e-api-b-3d38f6-74d6675f9d-sflzt\" (UID: \"46d350c3-9eff-403a-844d-f50edc815b45\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3d38f6-74d6675f9d-sflzt"
Oct 30 22:31:39.577214 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:39.577179    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6gdm\" (UniqueName: \"kubernetes.io/projected/46d350c3-9eff-403a-844d-f50edc815b45-kube-api-access-q6gdm\") pod \"maas-api-e2e-api-b-3d38f6-74d6675f9d-sflzt\" (UID: \"46d350c3-9eff-403a-844d-f50edc815b45\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3d38f6-74d6675f9d-sflzt"
Oct 30 22:31:40.062761 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:40.062721    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46d350c3-9eff-403a-844d-f50edc815b45-maas-api-tls\") pod \"maas-api-e2e-api-b-3d38f6-74d6675f9d-sflzt\" (UID: \"46d350c3-9eff-403a-844d-f50edc815b45\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3d38f6-74d6675f9d-sflzt"
Oct 30 22:31:40.062940 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:31:40.062896    4477 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-3d38f6-serving-cert: secret "maas-api-e2e-api-b-3d38f6-serving-cert" not found
Oct 30 22:31:40.063001 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:31:40.062988    4477 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46d350c3-9eff-403a-844d-f50edc815b45-maas-api-tls podName:46d350c3-9eff-403a-844d-f50edc815b45 nodeName:}" failed. No retries permitted until 2026-10-30 22:31:41.062967726 +0000 UTC m=+1484.988893930 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/46d350c3-9eff-403a-844d-f50edc815b45-maas-api-tls") pod "maas-api-e2e-api-b-3d38f6-74d6675f9d-sflzt" (UID: "46d350c3-9eff-403a-844d-f50edc815b45") : secret "maas-api-e2e-api-b-3d38f6-serving-cert" not found
Oct 30 22:31:40.590448 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:40.590416    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-2949c143/e2e-lifecycle-model-2949c143-kserve-74f9489d85-g9btm"]
Oct 30 22:31:41.078243 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:41.078205    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46d350c3-9eff-403a-844d-f50edc815b45-maas-api-tls\") pod \"maas-api-e2e-api-b-3d38f6-74d6675f9d-sflzt\" (UID: \"46d350c3-9eff-403a-844d-f50edc815b45\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3d38f6-74d6675f9d-sflzt"
Oct 30 22:31:41.081293 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:41.081261    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46d350c3-9eff-403a-844d-f50edc815b45-maas-api-tls\") pod \"maas-api-e2e-api-b-3d38f6-74d6675f9d-sflzt\" (UID: \"46d350c3-9eff-403a-844d-f50edc815b45\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3d38f6-74d6675f9d-sflzt"
Oct 30 22:31:41.355186 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:41.355074    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3d38f6-74d6675f9d-sflzt"
Oct 30 22:31:41.662697 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:41.662667    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-3d38f6-74d6675f9d-sflzt"]
Oct 30 22:31:41.664690 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:31:41.664659    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46d350c3_9eff_403a_844d_f50edc815b45.slice/crio-c57ec7ba044e6ff7ccbd291a48aa3ed81ec24a0ffd6b855838e033d9e92cf1ac WatchSource:0}: Error finding container c57ec7ba044e6ff7ccbd291a48aa3ed81ec24a0ffd6b855838e033d9e92cf1ac: Status 404 returned error can't find the container with id c57ec7ba044e6ff7ccbd291a48aa3ed81ec24a0ffd6b855838e033d9e92cf1ac
Oct 30 22:31:42.218555 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:42.218473    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3d38f6-74d6675f9d-sflzt" event={"ID":"46d350c3-9eff-403a-844d-f50edc815b45","Type":"ContainerStarted","Data":"db425b478f13991d695f8d1ac5424aef32358ea4c399c95d080e8bc0b9675960"}
Oct 30 22:31:42.218555 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:42.218539    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3d38f6-74d6675f9d-sflzt"
Oct 30 22:31:42.218801 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:42.218564    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3d38f6-74d6675f9d-sflzt" event={"ID":"46d350c3-9eff-403a-844d-f50edc815b45","Type":"ContainerStarted","Data":"c57ec7ba044e6ff7ccbd291a48aa3ed81ec24a0ffd6b855838e033d9e92cf1ac"}
Oct 30 22:31:42.258136 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:42.258076    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3d38f6-74d6675f9d-sflzt" podStartSLOduration=2.980234085 podStartE2EDuration="3.258063124s" totalImagesPullingTime="277.829039ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:31:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:31:42.255182508 +0000 UTC m=+1486.181108743" watchObservedRunningTime="2026-10-30 22:31:42.258063124 +0000 UTC m=+1486.183989324"
Oct 30 22:31:48.233136 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:48.233109    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3d38f6-74d6675f9d-sflzt"
Oct 30 22:31:54.414543 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:54.414509    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-7m76p"]
Oct 30 22:31:54.414947 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:54.414712    4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-7m76p" podUID="0477d090-6e4c-4599-b4e9-b8e103c67fd1" containerName="istio-proxy" containerID="cri-o://d12a0d10159da837f2cd37c125415bf1d92195a8858f12b0378a995bd11e3fbc" gracePeriod=30
Oct 30 22:31:54.657372 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:54.657341    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-rpk4x"]
Oct 30 22:31:54.657610 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:54.657582    4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-rpk4x" podUID="69b42852-cbc4-46b7-be1b-98ce37efa0d4" containerName="istio-proxy" containerID="cri-o://bba4ec6d91fcb18b01f95250806d3518a03ba22e431e61012fed79f81201a618" gracePeriod=30
Oct 30 22:31:55.617722 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:55.617663    4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-2949c143/e2e-lifecycle-model-2949c143-kserve-74f9489d85-g9btm" podUID="c0883afa-5295-4347-848e-ebfd7f831844" containerName="main" containerID="cri-o://0d95da8c6195f34e4d0fa8605e44efb3426ea2caab7a7e491126743f9edb41e9" gracePeriod=45
Oct 30 22:31:55.890627 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:55.890601    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-2949c143/e2e-lifecycle-model-2949c143-kserve-74f9489d85-g9btm"
Oct 30 22:31:55.983412 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:55.983376    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0883afa-5295-4347-848e-ebfd7f831844-home\" (UniqueName: \"kubernetes.io/empty-dir/c0883afa-5295-4347-848e-ebfd7f831844-home\") pod \"c0883afa-5295-4347-848e-ebfd7f831844\" (UID: \"c0883afa-5295-4347-848e-ebfd7f831844\") "
Oct 30 22:31:55.983624 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:55.983603    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0883afa-5295-4347-848e-ebfd7f831844-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0883afa-5295-4347-848e-ebfd7f831844-dshm\") pod \"c0883afa-5295-4347-848e-ebfd7f831844\" (UID: \"c0883afa-5295-4347-848e-ebfd7f831844\") "
Oct 30 22:31:55.983719 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:55.983681    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0883afa-5295-4347-848e-ebfd7f831844-home" pod "c0883afa-5295-4347-848e-ebfd7f831844" (UID: "c0883afa-5295-4347-848e-ebfd7f831844"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:55.983786 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:55.983692    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0883afa-5295-4347-848e-ebfd7f831844-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0883afa-5295-4347-848e-ebfd7f831844-tmp-dir\") pod \"c0883afa-5295-4347-848e-ebfd7f831844\" (UID: \"c0883afa-5295-4347-848e-ebfd7f831844\") "
Oct 30 22:31:55.983845 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:55.983811    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0883afa-5295-4347-848e-ebfd7f831844-tls-certs\" (UniqueName: \"kubernetes.io/secret/c0883afa-5295-4347-848e-ebfd7f831844-tls-certs\") pod \"c0883afa-5295-4347-848e-ebfd7f831844\" (UID: \"c0883afa-5295-4347-848e-ebfd7f831844\") "
Oct 30 22:31:55.983950 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:55.983929    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0883afa-5295-4347-848e-ebfd7f831844-kube-api-access-52xbc\" (UniqueName: \"kubernetes.io/projected/c0883afa-5295-4347-848e-ebfd7f831844-kube-api-access-52xbc\") pod \"c0883afa-5295-4347-848e-ebfd7f831844\" (UID: \"c0883afa-5295-4347-848e-ebfd7f831844\") "
Oct 30 22:31:55.984023 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:55.984008    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0883afa-5295-4347-848e-ebfd7f831844-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0883afa-5295-4347-848e-ebfd7f831844-model-cache\") pod \"c0883afa-5295-4347-848e-ebfd7f831844\" (UID: \"c0883afa-5295-4347-848e-ebfd7f831844\") "
Oct 30 22:31:55.984265 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:55.983926    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0883afa-5295-4347-848e-ebfd7f831844-tmp-dir" pod "c0883afa-5295-4347-848e-ebfd7f831844" (UID: "c0883afa-5295-4347-848e-ebfd7f831844"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:55.984578 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:55.984544    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0883afa-5295-4347-848e-ebfd7f831844-model-cache" pod "c0883afa-5295-4347-848e-ebfd7f831844" (UID: "c0883afa-5295-4347-848e-ebfd7f831844"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:55.985001 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:55.984977    4477 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0883afa-5295-4347-848e-ebfd7f831844-tmp-dir\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:31:55.985110 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:55.985006    4477 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0883afa-5295-4347-848e-ebfd7f831844-model-cache\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:31:55.985110 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:55.985020    4477 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0883afa-5295-4347-848e-ebfd7f831844-home\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:31:55.986522 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:55.986494    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0883afa-5295-4347-848e-ebfd7f831844-dshm" pod "c0883afa-5295-4347-848e-ebfd7f831844" (UID: "c0883afa-5295-4347-848e-ebfd7f831844"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:55.986948 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:55.986923    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0883afa-5295-4347-848e-ebfd7f831844-tls-certs" pod "c0883afa-5295-4347-848e-ebfd7f831844" (UID: "c0883afa-5295-4347-848e-ebfd7f831844"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:31:55.987213 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:55.987180    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0883afa-5295-4347-848e-ebfd7f831844-kube-api-access-52xbc" pod "c0883afa-5295-4347-848e-ebfd7f831844" (UID: "c0883afa-5295-4347-848e-ebfd7f831844"). InnerVolumeSpecName "kube-api-access-52xbc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:31:56.086484 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:56.086448    4477 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0883afa-5295-4347-848e-ebfd7f831844-dshm\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:31:56.086484 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:56.086479    4477 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0883afa-5295-4347-848e-ebfd7f831844-tls-certs\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:31:56.086484 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:56.086491    4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52xbc\" (UniqueName: \"kubernetes.io/projected/c0883afa-5295-4347-848e-ebfd7f831844-kube-api-access-52xbc\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:31:56.268493 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:31:56.268458    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 22:31:56.312987 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:56.312957    4477 generic.go:356] "Generic (PLEG): container finished" podID="c0883afa-5295-4347-848e-ebfd7f831844" containerID="0d95da8c6195f34e4d0fa8605e44efb3426ea2caab7a7e491126743f9edb41e9" exitCode=0
Oct 30 22:31:56.313156 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:56.312997    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-2949c143/e2e-lifecycle-model-2949c143-kserve-74f9489d85-g9btm"
Oct 30 22:31:56.313156 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:56.313053    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-2949c143/e2e-lifecycle-model-2949c143-kserve-74f9489d85-g9btm" event={"ID":"c0883afa-5295-4347-848e-ebfd7f831844","Type":"ContainerDied","Data":"0d95da8c6195f34e4d0fa8605e44efb3426ea2caab7a7e491126743f9edb41e9"}
Oct 30 22:31:56.313156 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:56.313105    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-2949c143/e2e-lifecycle-model-2949c143-kserve-74f9489d85-g9btm" event={"ID":"c0883afa-5295-4347-848e-ebfd7f831844","Type":"ContainerDied","Data":"30a3ddf7b4dbffccbec81afde71bdc8a47ef580b0c2e99bcfdef67403cfad566"}
Oct 30 22:31:56.313156 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:56.313127    4477 scope.go:122] "RemoveContainer" containerID="0d95da8c6195f34e4d0fa8605e44efb3426ea2caab7a7e491126743f9edb41e9"
Oct 30 22:31:56.322741 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:56.322719    4477 scope.go:122] "RemoveContainer" containerID="0d95da8c6195f34e4d0fa8605e44efb3426ea2caab7a7e491126743f9edb41e9"
Oct 30 22:31:56.323004 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:31:56.322987    4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d95da8c6195f34e4d0fa8605e44efb3426ea2caab7a7e491126743f9edb41e9\": container with ID starting with 0d95da8c6195f34e4d0fa8605e44efb3426ea2caab7a7e491126743f9edb41e9 not found: ID does not exist" containerID="0d95da8c6195f34e4d0fa8605e44efb3426ea2caab7a7e491126743f9edb41e9"
Oct 30 22:31:56.323068 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:56.323012    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d95da8c6195f34e4d0fa8605e44efb3426ea2caab7a7e491126743f9edb41e9"} err="failed to get container status \"0d95da8c6195f34e4d0fa8605e44efb3426ea2caab7a7e491126743f9edb41e9\": rpc error: code = NotFound desc = could not find container \"0d95da8c6195f34e4d0fa8605e44efb3426ea2caab7a7e491126743f9edb41e9\": container with ID starting with 0d95da8c6195f34e4d0fa8605e44efb3426ea2caab7a7e491126743f9edb41e9 not found: ID does not exist"
Oct 30 22:31:56.360870 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:56.360844    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-2949c143/e2e-lifecycle-model-2949c143-kserve-74f9489d85-g9btm"]
Oct 30 22:31:56.371456 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:56.371433    4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-2949c143/e2e-lifecycle-model-2949c143-kserve-74f9489d85-g9btm"]
Oct 30 22:31:56.815619 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:56.815600    4477 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jcf8l_015f218d-bf1f-4e7e-bb5d-b805af003246/ovn-acl-logging/0.log"
Oct 30 22:31:56.864793 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:56.864771    4477 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jcf8l_015f218d-bf1f-4e7e-bb5d-b805af003246/ovn-acl-logging/0.log"
Oct 30 22:31:58.197411 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:58.197373    4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0883afa-5295-4347-848e-ebfd7f831844" path="/var/lib/kubelet/pods/c0883afa-5295-4347-848e-ebfd7f831844/volumes"
Oct 30 22:31:59.798200 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:59.798176    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-7m76p"
Oct 30 22:31:59.873855 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:59.873819    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0477d090-6e4c-4599-b4e9-b8e103c67fd1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0477d090-6e4c-4599-b4e9-b8e103c67fd1-credential-socket\") pod \"0477d090-6e4c-4599-b4e9-b8e103c67fd1\" (UID: \"0477d090-6e4c-4599-b4e9-b8e103c67fd1\") "
Oct 30 22:31:59.874010 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:59.873938    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0477d090-6e4c-4599-b4e9-b8e103c67fd1-istio-token\" (UniqueName: \"kubernetes.io/projected/0477d090-6e4c-4599-b4e9-b8e103c67fd1-istio-token\") pod \"0477d090-6e4c-4599-b4e9-b8e103c67fd1\" (UID: \"0477d090-6e4c-4599-b4e9-b8e103c67fd1\") "
Oct 30 22:31:59.874084 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:59.874027    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0477d090-6e4c-4599-b4e9-b8e103c67fd1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0477d090-6e4c-4599-b4e9-b8e103c67fd1-istio-podinfo\") pod \"0477d090-6e4c-4599-b4e9-b8e103c67fd1\" (UID: \"0477d090-6e4c-4599-b4e9-b8e103c67fd1\") "
Oct 30 22:31:59.874133 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:59.874090    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0477d090-6e4c-4599-b4e9-b8e103c67fd1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0477d090-6e4c-4599-b4e9-b8e103c67fd1-istiod-ca-cert\") pod \"0477d090-6e4c-4599-b4e9-b8e103c67fd1\" (UID: \"0477d090-6e4c-4599-b4e9-b8e103c67fd1\") "
Oct 30 22:31:59.874204 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:59.874175    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0477d090-6e4c-4599-b4e9-b8e103c67fd1-credential-socket" pod "0477d090-6e4c-4599-b4e9-b8e103c67fd1" (UID: "0477d090-6e4c-4599-b4e9-b8e103c67fd1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:59.874204 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:59.874194    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0477d090-6e4c-4599-b4e9-b8e103c67fd1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0477d090-6e4c-4599-b4e9-b8e103c67fd1-istio-data\") pod \"0477d090-6e4c-4599-b4e9-b8e103c67fd1\" (UID: \"0477d090-6e4c-4599-b4e9-b8e103c67fd1\") "
Oct 30 22:31:59.874275 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:59.874256    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0477d090-6e4c-4599-b4e9-b8e103c67fd1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0477d090-6e4c-4599-b4e9-b8e103c67fd1-workload-certs\") pod \"0477d090-6e4c-4599-b4e9-b8e103c67fd1\" (UID: \"0477d090-6e4c-4599-b4e9-b8e103c67fd1\") "
Oct 30 22:31:59.874311 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:59.874289    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0477d090-6e4c-4599-b4e9-b8e103c67fd1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0477d090-6e4c-4599-b4e9-b8e103c67fd1-istio-envoy\") pod \"0477d090-6e4c-4599-b4e9-b8e103c67fd1\" (UID: \"0477d090-6e4c-4599-b4e9-b8e103c67fd1\") "
Oct 30 22:31:59.874349 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:59.874338    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0477d090-6e4c-4599-b4e9-b8e103c67fd1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0477d090-6e4c-4599-b4e9-b8e103c67fd1-workload-socket\") pod \"0477d090-6e4c-4599-b4e9-b8e103c67fd1\" (UID: \"0477d090-6e4c-4599-b4e9-b8e103c67fd1\") "
Oct 30 22:31:59.874402 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:59.874386    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0477d090-6e4c-4599-b4e9-b8e103c67fd1-kube-api-access-sqwkh\" (UniqueName: \"kubernetes.io/projected/0477d090-6e4c-4599-b4e9-b8e103c67fd1-kube-api-access-sqwkh\") pod \"0477d090-6e4c-4599-b4e9-b8e103c67fd1\" (UID: \"0477d090-6e4c-4599-b4e9-b8e103c67fd1\") "
Oct 30 22:31:59.874867 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:59.874679    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0477d090-6e4c-4599-b4e9-b8e103c67fd1-workload-certs" pod "0477d090-6e4c-4599-b4e9-b8e103c67fd1" (UID: "0477d090-6e4c-4599-b4e9-b8e103c67fd1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:59.875317 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:59.875118    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0477d090-6e4c-4599-b4e9-b8e103c67fd1-istio-data" pod "0477d090-6e4c-4599-b4e9-b8e103c67fd1" (UID: "0477d090-6e4c-4599-b4e9-b8e103c67fd1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:59.875317 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:59.875214    4477 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0477d090-6e4c-4599-b4e9-b8e103c67fd1-workload-certs\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:31:59.875317 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:59.875239    4477 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0477d090-6e4c-4599-b4e9-b8e103c67fd1-credential-socket\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:31:59.875529 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:59.875329    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0477d090-6e4c-4599-b4e9-b8e103c67fd1-workload-socket" pod "0477d090-6e4c-4599-b4e9-b8e103c67fd1" (UID: "0477d090-6e4c-4599-b4e9-b8e103c67fd1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:59.875798 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:59.875771    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0477d090-6e4c-4599-b4e9-b8e103c67fd1-istiod-ca-cert" pod "0477d090-6e4c-4599-b4e9-b8e103c67fd1" (UID: "0477d090-6e4c-4599-b4e9-b8e103c67fd1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:31:59.876995 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:59.876966    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0477d090-6e4c-4599-b4e9-b8e103c67fd1-istio-podinfo" pod "0477d090-6e4c-4599-b4e9-b8e103c67fd1" (UID: "0477d090-6e4c-4599-b4e9-b8e103c67fd1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:31:59.877980 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:59.877948    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0477d090-6e4c-4599-b4e9-b8e103c67fd1-istio-token" pod "0477d090-6e4c-4599-b4e9-b8e103c67fd1" (UID: "0477d090-6e4c-4599-b4e9-b8e103c67fd1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:31:59.879308 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:59.879285    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0477d090-6e4c-4599-b4e9-b8e103c67fd1-istio-envoy" pod "0477d090-6e4c-4599-b4e9-b8e103c67fd1" (UID: "0477d090-6e4c-4599-b4e9-b8e103c67fd1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:59.879405 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:59.879317    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0477d090-6e4c-4599-b4e9-b8e103c67fd1-kube-api-access-sqwkh" pod "0477d090-6e4c-4599-b4e9-b8e103c67fd1" (UID: "0477d090-6e4c-4599-b4e9-b8e103c67fd1"). InnerVolumeSpecName "kube-api-access-sqwkh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:31:59.921537 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:59.921517    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-rpk4x"
Oct 30 22:31:59.975822 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:59.975713    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69b42852-cbc4-46b7-be1b-98ce37efa0d4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69b42852-cbc4-46b7-be1b-98ce37efa0d4-credential-socket\") pod \"69b42852-cbc4-46b7-be1b-98ce37efa0d4\" (UID: \"69b42852-cbc4-46b7-be1b-98ce37efa0d4\") "
Oct 30 22:31:59.975822 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:59.975799    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69b42852-cbc4-46b7-be1b-98ce37efa0d4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69b42852-cbc4-46b7-be1b-98ce37efa0d4-istio-data\") pod \"69b42852-cbc4-46b7-be1b-98ce37efa0d4\" (UID: \"69b42852-cbc4-46b7-be1b-98ce37efa0d4\") "
Oct 30 22:31:59.976118 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:59.975855    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69b42852-cbc4-46b7-be1b-98ce37efa0d4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69b42852-cbc4-46b7-be1b-98ce37efa0d4-workload-certs\") pod \"69b42852-cbc4-46b7-be1b-98ce37efa0d4\" (UID: \"69b42852-cbc4-46b7-be1b-98ce37efa0d4\") "
Oct 30 22:31:59.976118 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:59.975974    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69b42852-cbc4-46b7-be1b-98ce37efa0d4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69b42852-cbc4-46b7-be1b-98ce37efa0d4-istiod-ca-cert\") pod \"69b42852-cbc4-46b7-be1b-98ce37efa0d4\" (UID: \"69b42852-cbc4-46b7-be1b-98ce37efa0d4\") "
Oct 30 22:31:59.976118 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:59.976017    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69b42852-cbc4-46b7-be1b-98ce37efa0d4-credential-socket" pod "69b42852-cbc4-46b7-be1b-98ce37efa0d4" (UID: "69b42852-cbc4-46b7-be1b-98ce37efa0d4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:59.976118 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:59.976020    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69b42852-cbc4-46b7-be1b-98ce37efa0d4-kube-api-access-6kvcz\" (UniqueName: \"kubernetes.io/projected/69b42852-cbc4-46b7-be1b-98ce37efa0d4-kube-api-access-6kvcz\") pod \"69b42852-cbc4-46b7-be1b-98ce37efa0d4\" (UID: \"69b42852-cbc4-46b7-be1b-98ce37efa0d4\") "
Oct 30 22:31:59.976357 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:59.976161    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69b42852-cbc4-46b7-be1b-98ce37efa0d4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69b42852-cbc4-46b7-be1b-98ce37efa0d4-workload-socket\") pod \"69b42852-cbc4-46b7-be1b-98ce37efa0d4\" (UID: \"69b42852-cbc4-46b7-be1b-98ce37efa0d4\") "
Oct 30 22:31:59.976357 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:59.976165    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69b42852-cbc4-46b7-be1b-98ce37efa0d4-istio-data" pod "69b42852-cbc4-46b7-be1b-98ce37efa0d4" (UID: "69b42852-cbc4-46b7-be1b-98ce37efa0d4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:59.976357 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:59.976189    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69b42852-cbc4-46b7-be1b-98ce37efa0d4-workload-certs" pod "69b42852-cbc4-46b7-be1b-98ce37efa0d4" (UID: "69b42852-cbc4-46b7-be1b-98ce37efa0d4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:59.976357 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:59.976260    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69b42852-cbc4-46b7-be1b-98ce37efa0d4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69b42852-cbc4-46b7-be1b-98ce37efa0d4-istio-envoy\") pod \"69b42852-cbc4-46b7-be1b-98ce37efa0d4\" (UID: \"69b42852-cbc4-46b7-be1b-98ce37efa0d4\") "
Oct 30 22:31:59.976357 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:59.976313    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/69b42852-cbc4-46b7-be1b-98ce37efa0d4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69b42852-cbc4-46b7-be1b-98ce37efa0d4-istio-podinfo\") pod \"69b42852-cbc4-46b7-be1b-98ce37efa0d4\" (UID: \"69b42852-cbc4-46b7-be1b-98ce37efa0d4\") "
Oct 30 22:31:59.976357 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:59.976351    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69b42852-cbc4-46b7-be1b-98ce37efa0d4-istio-token\" (UniqueName: \"kubernetes.io/projected/69b42852-cbc4-46b7-be1b-98ce37efa0d4-istio-token\") pod \"69b42852-cbc4-46b7-be1b-98ce37efa0d4\" (UID: \"69b42852-cbc4-46b7-be1b-98ce37efa0d4\") "
Oct 30 22:31:59.976667 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:59.976468    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69b42852-cbc4-46b7-be1b-98ce37efa0d4-workload-socket" pod "69b42852-cbc4-46b7-be1b-98ce37efa0d4" (UID: "69b42852-cbc4-46b7-be1b-98ce37efa0d4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:59.976667 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:59.976506    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69b42852-cbc4-46b7-be1b-98ce37efa0d4-istiod-ca-cert" pod "69b42852-cbc4-46b7-be1b-98ce37efa0d4" (UID: "69b42852-cbc4-46b7-be1b-98ce37efa0d4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:31:59.977144 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:59.977119    4477 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0477d090-6e4c-4599-b4e9-b8e103c67fd1-istio-envoy\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:31:59.977265 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:59.977146    4477 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0477d090-6e4c-4599-b4e9-b8e103c67fd1-workload-socket\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:31:59.977265 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:59.977162    4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqwkh\" (UniqueName: \"kubernetes.io/projected/0477d090-6e4c-4599-b4e9-b8e103c67fd1-kube-api-access-sqwkh\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:31:59.977265 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:59.977178    4477 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69b42852-cbc4-46b7-be1b-98ce37efa0d4-istiod-ca-cert\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:31:59.977265 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:59.977192    4477 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69b42852-cbc4-46b7-be1b-98ce37efa0d4-workload-socket\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:31:59.977265 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:59.977205    4477 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0477d090-6e4c-4599-b4e9-b8e103c67fd1-istio-token\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:31:59.977265 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:59.977218    4477 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69b42852-cbc4-46b7-be1b-98ce37efa0d4-credential-socket\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:31:59.977265 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:59.977232    4477 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0477d090-6e4c-4599-b4e9-b8e103c67fd1-istio-podinfo\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:31:59.977265 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:59.977246    4477 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0477d090-6e4c-4599-b4e9-b8e103c67fd1-istiod-ca-cert\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:31:59.977265 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:59.977257    4477 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69b42852-cbc4-46b7-be1b-98ce37efa0d4-istio-data\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:31:59.977708 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:59.977270    4477 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69b42852-cbc4-46b7-be1b-98ce37efa0d4-workload-certs\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:31:59.977708 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:59.977282    4477 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0477d090-6e4c-4599-b4e9-b8e103c67fd1-istio-data\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:31:59.979134 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:59.979084    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/69b42852-cbc4-46b7-be1b-98ce37efa0d4-istio-podinfo" pod "69b42852-cbc4-46b7-be1b-98ce37efa0d4" (UID: "69b42852-cbc4-46b7-be1b-98ce37efa0d4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:31:59.979242 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:59.979136    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69b42852-cbc4-46b7-be1b-98ce37efa0d4-istio-envoy" pod "69b42852-cbc4-46b7-be1b-98ce37efa0d4" (UID: "69b42852-cbc4-46b7-be1b-98ce37efa0d4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:31:59.979242 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:59.979140    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69b42852-cbc4-46b7-be1b-98ce37efa0d4-istio-token" pod "69b42852-cbc4-46b7-be1b-98ce37efa0d4" (UID: "69b42852-cbc4-46b7-be1b-98ce37efa0d4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:31:59.979359 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:31:59.979332    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69b42852-cbc4-46b7-be1b-98ce37efa0d4-kube-api-access-6kvcz" pod "69b42852-cbc4-46b7-be1b-98ce37efa0d4" (UID: "69b42852-cbc4-46b7-be1b-98ce37efa0d4"). InnerVolumeSpecName "kube-api-access-6kvcz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:32:00.081443 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:32:00.080241    4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kvcz\" (UniqueName: \"kubernetes.io/projected/69b42852-cbc4-46b7-be1b-98ce37efa0d4-kube-api-access-6kvcz\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:32:00.081443 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:32:00.080280    4477 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69b42852-cbc4-46b7-be1b-98ce37efa0d4-istio-envoy\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:32:00.081443 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:32:00.080295    4477 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69b42852-cbc4-46b7-be1b-98ce37efa0d4-istio-podinfo\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:32:00.081443 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:32:00.080307    4477 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69b42852-cbc4-46b7-be1b-98ce37efa0d4-istio-token\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:32:00.358798 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:32:00.358712    4477 generic.go:356] "Generic (PLEG): container finished" podID="0477d090-6e4c-4599-b4e9-b8e103c67fd1" containerID="d12a0d10159da837f2cd37c125415bf1d92195a8858f12b0378a995bd11e3fbc" exitCode=0
Oct 30 22:32:00.358798 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:32:00.358751    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-7m76p"
Oct 30 22:32:00.359032 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:32:00.358794    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-7m76p" event={"ID":"0477d090-6e4c-4599-b4e9-b8e103c67fd1","Type":"ContainerDied","Data":"d12a0d10159da837f2cd37c125415bf1d92195a8858f12b0378a995bd11e3fbc"}
Oct 30 22:32:00.359032 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:32:00.358860    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-7m76p" event={"ID":"0477d090-6e4c-4599-b4e9-b8e103c67fd1","Type":"ContainerDied","Data":"41f4023bfce3e7503666f38e3ecd1f18b70b7e5ad7122b4f6359c656f53b537a"}
Oct 30 22:32:00.359032 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:32:00.358890    4477 scope.go:122] "RemoveContainer" containerID="d12a0d10159da837f2cd37c125415bf1d92195a8858f12b0378a995bd11e3fbc"
Oct 30 22:32:00.360863 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:32:00.360842    4477 generic.go:356] "Generic (PLEG): container finished" podID="69b42852-cbc4-46b7-be1b-98ce37efa0d4" containerID="bba4ec6d91fcb18b01f95250806d3518a03ba22e431e61012fed79f81201a618" exitCode=0
Oct 30 22:32:00.360989 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:32:00.360891    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-rpk4x"
Oct 30 22:32:00.360989 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:32:00.360880    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-rpk4x" event={"ID":"69b42852-cbc4-46b7-be1b-98ce37efa0d4","Type":"ContainerDied","Data":"bba4ec6d91fcb18b01f95250806d3518a03ba22e431e61012fed79f81201a618"}
Oct 30 22:32:00.360989 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:32:00.360959    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-rpk4x" event={"ID":"69b42852-cbc4-46b7-be1b-98ce37efa0d4","Type":"ContainerDied","Data":"3645df837bb85026d887dc8af7079c5233c325eb490b5de0187f0d3719d445f7"}
Oct 30 22:32:00.376036 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:32:00.376015    4477 scope.go:122] "RemoveContainer" containerID="d12a0d10159da837f2cd37c125415bf1d92195a8858f12b0378a995bd11e3fbc"
Oct 30 22:32:00.376423 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:32:00.376402    4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d12a0d10159da837f2cd37c125415bf1d92195a8858f12b0378a995bd11e3fbc\": container with ID starting with d12a0d10159da837f2cd37c125415bf1d92195a8858f12b0378a995bd11e3fbc not found: ID does not exist" containerID="d12a0d10159da837f2cd37c125415bf1d92195a8858f12b0378a995bd11e3fbc"
Oct 30 22:32:00.376477 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:32:00.376433    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d12a0d10159da837f2cd37c125415bf1d92195a8858f12b0378a995bd11e3fbc"} err="failed to get container status \"d12a0d10159da837f2cd37c125415bf1d92195a8858f12b0378a995bd11e3fbc\": rpc error: code = NotFound desc = could not find container \"d12a0d10159da837f2cd37c125415bf1d92195a8858f12b0378a995bd11e3fbc\": container with ID starting with d12a0d10159da837f2cd37c125415bf1d92195a8858f12b0378a995bd11e3fbc not found: ID does not exist"
Oct 30 22:32:00.376477 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:32:00.376453    4477 scope.go:122] "RemoveContainer" containerID="bba4ec6d91fcb18b01f95250806d3518a03ba22e431e61012fed79f81201a618"
Oct 30 22:32:00.388601 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:32:00.388581    4477 scope.go:122] "RemoveContainer" containerID="bba4ec6d91fcb18b01f95250806d3518a03ba22e431e61012fed79f81201a618"
Oct 30 22:32:00.388879 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:32:00.388862    4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bba4ec6d91fcb18b01f95250806d3518a03ba22e431e61012fed79f81201a618\": container with ID starting with bba4ec6d91fcb18b01f95250806d3518a03ba22e431e61012fed79f81201a618 not found: ID does not exist" containerID="bba4ec6d91fcb18b01f95250806d3518a03ba22e431e61012fed79f81201a618"
Oct 30 22:32:00.388932 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:32:00.388887    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bba4ec6d91fcb18b01f95250806d3518a03ba22e431e61012fed79f81201a618"} err="failed to get container status \"bba4ec6d91fcb18b01f95250806d3518a03ba22e431e61012fed79f81201a618\": rpc error: code = NotFound desc = could not find container \"bba4ec6d91fcb18b01f95250806d3518a03ba22e431e61012fed79f81201a618\": container with ID starting with bba4ec6d91fcb18b01f95250806d3518a03ba22e431e61012fed79f81201a618 not found: ID does not exist"
Oct 30 22:32:00.412181 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:32:00.412104    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-rpk4x"]
Oct 30 22:32:00.420597 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:32:00.420571    4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-rpk4x"]
Oct 30 22:32:00.450748 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:32:00.450723    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-7m76p"]
Oct 30 22:32:00.454376 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:32:00.454350    4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-7m76p"]
Oct 30 22:32:02.195008 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:32:02.194973    4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0477d090-6e4c-4599-b4e9-b8e103c67fd1" path="/var/lib/kubelet/pods/0477d090-6e4c-4599-b4e9-b8e103c67fd1/volumes"
Oct 30 22:32:02.195525 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:32:02.195509    4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69b42852-cbc4-46b7-be1b-98ce37efa0d4" path="/var/lib/kubelet/pods/69b42852-cbc4-46b7-be1b-98ce37efa0d4/volumes"
Oct 30 22:32:11.332424 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:32:11.332392    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8b3e43-68bdb79f49-9kn25"]
Oct 30 22:32:11.333106 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:32:11.333087    4477 cpu_manager.go:414] "containerMap: removing container" podUID="0477d090-6e4c-4599-b4e9-b8e103c67fd1" containerName="istio-proxy"
Oct 30 22:32:11.333106 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:32:11.333104    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0477d090-6e4c-4599-b4e9-b8e103c67fd1" containerName="istio-proxy"
Oct 30 22:32:11.333222 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:32:11.333115    4477 cpu_manager.go:414] "containerMap: removing container" podUID="c0883afa-5295-4347-848e-ebfd7f831844" containerName="main"
Oct 30 22:32:11.333222 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:32:11.333120    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0883afa-5295-4347-848e-ebfd7f831844" containerName="main"
Oct 30 22:32:11.333222 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:32:11.333133    4477 cpu_manager.go:414] "containerMap: removing container" podUID="69b42852-cbc4-46b7-be1b-98ce37efa0d4" containerName="istio-proxy"
Oct 30 22:32:11.333222 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:32:11.333138    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69b42852-cbc4-46b7-be1b-98ce37efa0d4" containerName="istio-proxy"
Oct 30 22:32:11.333414 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:32:11.333265    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="0477d090-6e4c-4599-b4e9-b8e103c67fd1" containerName="istio-proxy"
Oct 30 22:32:11.333414 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:32:11.333274    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0883afa-5295-4347-848e-ebfd7f831844" containerName="main"
Oct 30 22:32:11.333414 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:32:11.333281    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="69b42852-cbc4-46b7-be1b-98ce37efa0d4" containerName="istio-proxy"
Oct 30 22:32:11.338787 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:32:11.338764    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8b3e43-68bdb79f49-9kn25"
Oct 30 22:32:11.364019 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:32:11.363989    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-8b3e43-serving-cert\""
Oct 30 22:32:11.368168 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:32:11.368141    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8b3e43-68bdb79f49-9kn25"]
Oct 30 22:32:11.440416 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:32:11.440374    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b547204-65d4-42e4-83c3-914a7ee0e951-openshift-service-ca\") pod \"maas-api-e2e-api-b-8b3e43-68bdb79f49-9kn25\" (UID: \"0b547204-65d4-42e4-83c3-914a7ee0e951\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8b3e43-68bdb79f49-9kn25"
Oct 30 22:32:11.440579 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:32:11.440501    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b547204-65d4-42e4-83c3-914a7ee0e951-metrics-certs\") pod \"maas-api-e2e-api-b-8b3e43-68bdb79f49-9kn25\" (UID: \"0b547204-65d4-42e4-83c3-914a7ee0e951\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8b3e43-68bdb79f49-9kn25"
Oct 30 22:32:11.440579 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:32:11.440547    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b547204-65d4-42e4-83c3-914a7ee0e951-maas-api-tls\") pod \"maas-api-e2e-api-b-8b3e43-68bdb79f49-9kn25\" (UID: \"0b547204-65d4-42e4-83c3-914a7ee0e951\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8b3e43-68bdb79f49-9kn25"
Oct 30 22:32:11.440693 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:32:11.440679    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlxw4\" (UniqueName: \"kubernetes.io/projected/0b547204-65d4-42e4-83c3-914a7ee0e951-kube-api-access-vlxw4\") pod \"maas-api-e2e-api-b-8b3e43-68bdb79f49-9kn25\" (UID: \"0b547204-65d4-42e4-83c3-914a7ee0e951\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8b3e43-68bdb79f49-9kn25"
Oct 30 22:32:11.543344 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:32:11.542858    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b547204-65d4-42e4-83c3-914a7ee0e951-openshift-service-ca\") pod \"maas-api-e2e-api-b-8b3e43-68bdb79f49-9kn25\" (UID: \"0b547204-65d4-42e4-83c3-914a7ee0e951\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8b3e43-68bdb79f49-9kn25"
Oct 30 22:32:11.543344 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:32:11.542983    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b547204-65d4-42e4-83c3-914a7ee0e951-metrics-certs\") pod \"maas-api-e2e-api-b-8b3e43-68bdb79f49-9kn25\" (UID: \"0b547204-65d4-42e4-83c3-914a7ee0e951\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8b3e43-68bdb79f49-9kn25"
Oct 30 22:32:11.544234 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:32:11.544207    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b547204-65d4-42e4-83c3-914a7ee0e951-openshift-service-ca\") pod \"maas-api-e2e-api-b-8b3e43-68bdb79f49-9kn25\" (UID: \"0b547204-65d4-42e4-83c3-914a7ee0e951\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8b3e43-68bdb79f49-9kn25"
Oct 30 22:32:11.544402 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:32:11.544326    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b547204-65d4-42e4-83c3-914a7ee0e951-maas-api-tls\") pod \"maas-api-e2e-api-b-8b3e43-68bdb79f49-9kn25\" (UID: \"0b547204-65d4-42e4-83c3-914a7ee0e951\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8b3e43-68bdb79f49-9kn25"
Oct 30 22:32:11.544477 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:32:11.544455    4477 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-8b3e43-serving-cert: secret "maas-api-e2e-api-b-8b3e43-serving-cert" not found
Oct 30 22:32:11.544536 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:32:11.544489    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlxw4\" (UniqueName: \"kubernetes.io/projected/0b547204-65d4-42e4-83c3-914a7ee0e951-kube-api-access-vlxw4\") pod \"maas-api-e2e-api-b-8b3e43-68bdb79f49-9kn25\" (UID: \"0b547204-65d4-42e4-83c3-914a7ee0e951\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8b3e43-68bdb79f49-9kn25"
Oct 30 22:32:11.544536 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:32:11.544525    4477 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b547204-65d4-42e4-83c3-914a7ee0e951-maas-api-tls podName:0b547204-65d4-42e4-83c3-914a7ee0e951 nodeName:}" failed. No retries permitted until 2026-10-30 22:32:12.044509275 +0000 UTC m=+1515.970435459 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0b547204-65d4-42e4-83c3-914a7ee0e951-maas-api-tls") pod "maas-api-e2e-api-b-8b3e43-68bdb79f49-9kn25" (UID: "0b547204-65d4-42e4-83c3-914a7ee0e951") : secret "maas-api-e2e-api-b-8b3e43-serving-cert" not found
Oct 30 22:32:11.547296 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:32:11.547276    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b547204-65d4-42e4-83c3-914a7ee0e951-metrics-certs\") pod \"maas-api-e2e-api-b-8b3e43-68bdb79f49-9kn25\" (UID: \"0b547204-65d4-42e4-83c3-914a7ee0e951\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8b3e43-68bdb79f49-9kn25"
Oct 30 22:32:11.567168 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:32:11.567138    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlxw4\" (UniqueName: \"kubernetes.io/projected/0b547204-65d4-42e4-83c3-914a7ee0e951-kube-api-access-vlxw4\") pod \"maas-api-e2e-api-b-8b3e43-68bdb79f49-9kn25\" (UID: \"0b547204-65d4-42e4-83c3-914a7ee0e951\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8b3e43-68bdb79f49-9kn25"
Oct 30 22:32:12.051420 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:32:12.051387    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b547204-65d4-42e4-83c3-914a7ee0e951-maas-api-tls\") pod \"maas-api-e2e-api-b-8b3e43-68bdb79f49-9kn25\" (UID: \"0b547204-65d4-42e4-83c3-914a7ee0e951\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8b3e43-68bdb79f49-9kn25"
Oct 30 22:32:12.053887 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:32:12.053858    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b547204-65d4-42e4-83c3-914a7ee0e951-maas-api-tls\") pod \"maas-api-e2e-api-b-8b3e43-68bdb79f49-9kn25\" (UID: \"0b547204-65d4-42e4-83c3-914a7ee0e951\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8b3e43-68bdb79f49-9kn25"
Oct 30 22:32:12.252808 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:32:12.252779    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8b3e43-68bdb79f49-9kn25"
Oct 30 22:32:12.593885 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:32:12.593755    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8b3e43-68bdb79f49-9kn25"]
Oct 30 22:32:13.450224 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:32:13.450185    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8b3e43-68bdb79f49-9kn25" event={"ID":"0b547204-65d4-42e4-83c3-914a7ee0e951","Type":"ContainerStarted","Data":"faa9afe80a2de4322ee86d2bb6bd90cdf94cc4765234eb09174ebd4d2895294b"}
Oct 30 22:32:13.450425 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:32:13.450245    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8b3e43-68bdb79f49-9kn25" event={"ID":"0b547204-65d4-42e4-83c3-914a7ee0e951","Type":"ContainerStarted","Data":"e5ce2a38f4858205560de528200c9339ddfd5a1058c0bc11b3a37451d70a6779"}
Oct 30 22:32:13.450425 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:32:13.450269    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8b3e43-68bdb79f49-9kn25"
Oct 30 22:32:13.492825 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:32:13.492757    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8b3e43-68bdb79f49-9kn25" podStartSLOduration=2.162718204 podStartE2EDuration="2.492736449s" totalImagesPullingTime="330.018245ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:32:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:32:13.488645912 +0000 UTC m=+1517.414572120" watchObservedRunningTime="2026-10-30 22:32:13.492736449 +0000 UTC m=+1517.418662655"
Oct 30 22:32:19.463170 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:32:19.463140    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8b3e43-68bdb79f49-9kn25"
Oct 30 22:32:26.283926 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:32:26.283880    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 22:32:26.609521 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:32:26.609449    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6bz92"]
Oct 30 22:32:48.220167 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:32:48.220135    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4f7439-85d59f65c5-5j5r5"]
Oct 30 22:32:48.224417 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:32:48.224398    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4f7439-85d59f65c5-5j5r5"
Oct 30 22:32:48.237472 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:32:48.237453    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-4f7439-serving-cert\""
Oct 30 22:32:48.251352 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:32:48.251329    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4f7439-85d59f65c5-5j5r5"]
Oct 30 22:32:48.326001 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:32:48.325978    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36f8f834-bec7-411d-a4bc-86c141eecc14-metrics-certs\") pod \"maas-api-e2e-api-b-4f7439-85d59f65c5-5j5r5\" (UID: \"36f8f834-bec7-411d-a4bc-86c141eecc14\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4f7439-85d59f65c5-5j5r5"
Oct 30 22:32:48.326161 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:32:48.326135    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nw72p\" (UniqueName: \"kubernetes.io/projected/36f8f834-bec7-411d-a4bc-86c141eecc14-kube-api-access-nw72p\") pod \"maas-api-e2e-api-b-4f7439-85d59f65c5-5j5r5\" (UID: \"36f8f834-bec7-411d-a4bc-86c141eecc14\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4f7439-85d59f65c5-5j5r5"
Oct 30 22:32:48.326208 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:32:48.326193    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36f8f834-bec7-411d-a4bc-86c141eecc14-maas-api-tls\") pod \"maas-api-e2e-api-b-4f7439-85d59f65c5-5j5r5\" (UID: \"36f8f834-bec7-411d-a4bc-86c141eecc14\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4f7439-85d59f65c5-5j5r5"
Oct 30 22:32:48.326348 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:32:48.326331    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36f8f834-bec7-411d-a4bc-86c141eecc14-openshift-service-ca\") pod \"maas-api-e2e-api-b-4f7439-85d59f65c5-5j5r5\" (UID: \"36f8f834-bec7-411d-a4bc-86c141eecc14\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4f7439-85d59f65c5-5j5r5"
Oct 30 22:32:48.427856 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:32:48.427811    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36f8f834-bec7-411d-a4bc-86c141eecc14-openshift-service-ca\") pod \"maas-api-e2e-api-b-4f7439-85d59f65c5-5j5r5\" (UID: \"36f8f834-bec7-411d-a4bc-86c141eecc14\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4f7439-85d59f65c5-5j5r5"
Oct 30 22:32:48.428074 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:32:48.427965    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36f8f834-bec7-411d-a4bc-86c141eecc14-metrics-certs\") pod \"maas-api-e2e-api-b-4f7439-85d59f65c5-5j5r5\" (UID: \"36f8f834-bec7-411d-a4bc-86c141eecc14\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4f7439-85d59f65c5-5j5r5"
Oct 30 22:32:48.428164 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:32:48.428089    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nw72p\" (UniqueName: \"kubernetes.io/projected/36f8f834-bec7-411d-a4bc-86c141eecc14-kube-api-access-nw72p\") pod \"maas-api-e2e-api-b-4f7439-85d59f65c5-5j5r5\" (UID: \"36f8f834-bec7-411d-a4bc-86c141eecc14\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4f7439-85d59f65c5-5j5r5"
Oct 30 22:32:48.428224 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:32:48.428162    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36f8f834-bec7-411d-a4bc-86c141eecc14-maas-api-tls\") pod \"maas-api-e2e-api-b-4f7439-85d59f65c5-5j5r5\" (UID: \"36f8f834-bec7-411d-a4bc-86c141eecc14\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4f7439-85d59f65c5-5j5r5"
Oct 30 22:32:48.428397 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:32:48.428377    4477 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-4f7439-serving-cert: secret "maas-api-e2e-api-b-4f7439-serving-cert" not found
Oct 30 22:32:48.428484 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:32:48.428452    4477 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36f8f834-bec7-411d-a4bc-86c141eecc14-maas-api-tls podName:36f8f834-bec7-411d-a4bc-86c141eecc14 nodeName:}" failed. No retries permitted until 2026-10-30 22:32:48.928431358 +0000 UTC m=+1552.854357552 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/36f8f834-bec7-411d-a4bc-86c141eecc14-maas-api-tls") pod "maas-api-e2e-api-b-4f7439-85d59f65c5-5j5r5" (UID: "36f8f834-bec7-411d-a4bc-86c141eecc14") : secret "maas-api-e2e-api-b-4f7439-serving-cert" not found
Oct 30 22:32:48.428706 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:32:48.428671    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36f8f834-bec7-411d-a4bc-86c141eecc14-openshift-service-ca\") pod \"maas-api-e2e-api-b-4f7439-85d59f65c5-5j5r5\" (UID: \"36f8f834-bec7-411d-a4bc-86c141eecc14\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4f7439-85d59f65c5-5j5r5"
Oct 30 22:32:48.431085 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:32:48.431057    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36f8f834-bec7-411d-a4bc-86c141eecc14-metrics-certs\") pod \"maas-api-e2e-api-b-4f7439-85d59f65c5-5j5r5\" (UID: \"36f8f834-bec7-411d-a4bc-86c141eecc14\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4f7439-85d59f65c5-5j5r5"
Oct 30 22:32:48.449022 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:32:48.448995    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nw72p\" (UniqueName: \"kubernetes.io/projected/36f8f834-bec7-411d-a4bc-86c141eecc14-kube-api-access-nw72p\") pod \"maas-api-e2e-api-b-4f7439-85d59f65c5-5j5r5\" (UID: \"36f8f834-bec7-411d-a4bc-86c141eecc14\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4f7439-85d59f65c5-5j5r5"
Oct 30 22:32:48.935945 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:32:48.935913    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36f8f834-bec7-411d-a4bc-86c141eecc14-maas-api-tls\") pod \"maas-api-e2e-api-b-4f7439-85d59f65c5-5j5r5\" (UID: \"36f8f834-bec7-411d-a4bc-86c141eecc14\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4f7439-85d59f65c5-5j5r5"
Oct 30 22:32:48.938498 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:32:48.938470    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36f8f834-bec7-411d-a4bc-86c141eecc14-maas-api-tls\") pod \"maas-api-e2e-api-b-4f7439-85d59f65c5-5j5r5\" (UID: \"36f8f834-bec7-411d-a4bc-86c141eecc14\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4f7439-85d59f65c5-5j5r5"
Oct 30 22:32:49.138925 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:32:49.138887    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4f7439-85d59f65c5-5j5r5"
Oct 30 22:32:49.554993 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:32:49.554958    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4f7439-85d59f65c5-5j5r5"]
Oct 30 22:32:49.767870 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:32:49.767786    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4f7439-85d59f65c5-5j5r5" event={"ID":"36f8f834-bec7-411d-a4bc-86c141eecc14","Type":"ContainerStarted","Data":"4b872e6c0c74a39c0cdc704903b77270ca773f2c578fad8bf89a95457c71232f"}
Oct 30 22:32:50.776317 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:32:50.776281    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4f7439-85d59f65c5-5j5r5" event={"ID":"36f8f834-bec7-411d-a4bc-86c141eecc14","Type":"ContainerStarted","Data":"0a591696b32e0c890267bf149eb77fc8887c4444f0c9dde3ec63f522bc8c5296"}
Oct 30 22:32:50.776805 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:32:50.776372    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4f7439-85d59f65c5-5j5r5"
Oct 30 22:32:50.816253 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:32:50.816200    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4f7439-85d59f65c5-5j5r5" podStartSLOduration=2.493026165 podStartE2EDuration="2.816181728s" totalImagesPullingTime="323.155563ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:32:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:32:50.812435181 +0000 UTC m=+1554.738361388" watchObservedRunningTime="2026-10-30 22:32:50.816181728 +0000 UTC m=+1554.742107935"
Oct 30 22:32:56.271542 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:32:56.271505    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 22:32:56.788496 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:32:56.788462    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4f7439-85d59f65c5-5j5r5"
Oct 30 22:33:13.998411 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:13.998336    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-62236a-7fc8b485b4-tspl7"]
Oct 30 22:33:14.002961 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:14.002939    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-62236a-7fc8b485b4-tspl7"
Oct 30 22:33:14.029335 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:14.029274    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-62236a-serving-cert\""
Oct 30 22:33:14.034494 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:14.034465    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-62236a-7fc8b485b4-tspl7"]
Oct 30 22:33:14.060606 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:14.060574    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxdwk\" (UniqueName: \"kubernetes.io/projected/d5aad55c-1b94-460e-96e3-a40d87b421e0-kube-api-access-zxdwk\") pod \"maas-api-e2e-api-b-62236a-7fc8b485b4-tspl7\" (UID: \"d5aad55c-1b94-460e-96e3-a40d87b421e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-62236a-7fc8b485b4-tspl7"
Oct 30 22:33:14.060779 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:14.060627    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5aad55c-1b94-460e-96e3-a40d87b421e0-metrics-certs\") pod \"maas-api-e2e-api-b-62236a-7fc8b485b4-tspl7\" (UID: \"d5aad55c-1b94-460e-96e3-a40d87b421e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-62236a-7fc8b485b4-tspl7"
Oct 30 22:33:14.060881 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:14.060861    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5aad55c-1b94-460e-96e3-a40d87b421e0-maas-api-tls\") pod \"maas-api-e2e-api-b-62236a-7fc8b485b4-tspl7\" (UID: \"d5aad55c-1b94-460e-96e3-a40d87b421e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-62236a-7fc8b485b4-tspl7"
Oct 30 22:33:14.061150 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:14.061124    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5aad55c-1b94-460e-96e3-a40d87b421e0-openshift-service-ca\") pod \"maas-api-e2e-api-b-62236a-7fc8b485b4-tspl7\" (UID: \"d5aad55c-1b94-460e-96e3-a40d87b421e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-62236a-7fc8b485b4-tspl7"
Oct 30 22:33:14.163732 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:14.162770    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5aad55c-1b94-460e-96e3-a40d87b421e0-openshift-service-ca\") pod \"maas-api-e2e-api-b-62236a-7fc8b485b4-tspl7\" (UID: \"d5aad55c-1b94-460e-96e3-a40d87b421e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-62236a-7fc8b485b4-tspl7"
Oct 30 22:33:14.163732 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:14.162868    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxdwk\" (UniqueName: \"kubernetes.io/projected/d5aad55c-1b94-460e-96e3-a40d87b421e0-kube-api-access-zxdwk\") pod \"maas-api-e2e-api-b-62236a-7fc8b485b4-tspl7\" (UID: \"d5aad55c-1b94-460e-96e3-a40d87b421e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-62236a-7fc8b485b4-tspl7"
Oct 30 22:33:14.163732 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:14.162908    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5aad55c-1b94-460e-96e3-a40d87b421e0-metrics-certs\") pod \"maas-api-e2e-api-b-62236a-7fc8b485b4-tspl7\" (UID: \"d5aad55c-1b94-460e-96e3-a40d87b421e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-62236a-7fc8b485b4-tspl7"
Oct 30 22:33:14.163732 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:14.163020    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5aad55c-1b94-460e-96e3-a40d87b421e0-maas-api-tls\") pod \"maas-api-e2e-api-b-62236a-7fc8b485b4-tspl7\" (UID: \"d5aad55c-1b94-460e-96e3-a40d87b421e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-62236a-7fc8b485b4-tspl7"
Oct 30 22:33:14.163732 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:33:14.163316    4477 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-62236a-serving-cert: secret "maas-api-e2e-api-b-62236a-serving-cert" not found
Oct 30 22:33:14.163732 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:33:14.163429    4477 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5aad55c-1b94-460e-96e3-a40d87b421e0-maas-api-tls podName:d5aad55c-1b94-460e-96e3-a40d87b421e0 nodeName:}" failed. No retries permitted until 2026-10-30 22:33:14.663395493 +0000 UTC m=+1578.589321679 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d5aad55c-1b94-460e-96e3-a40d87b421e0-maas-api-tls") pod "maas-api-e2e-api-b-62236a-7fc8b485b4-tspl7" (UID: "d5aad55c-1b94-460e-96e3-a40d87b421e0") : secret "maas-api-e2e-api-b-62236a-serving-cert" not found
Oct 30 22:33:14.168061 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:14.164463    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5aad55c-1b94-460e-96e3-a40d87b421e0-openshift-service-ca\") pod \"maas-api-e2e-api-b-62236a-7fc8b485b4-tspl7\" (UID: \"d5aad55c-1b94-460e-96e3-a40d87b421e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-62236a-7fc8b485b4-tspl7"
Oct 30 22:33:14.168061 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:14.166893    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5aad55c-1b94-460e-96e3-a40d87b421e0-metrics-certs\") pod \"maas-api-e2e-api-b-62236a-7fc8b485b4-tspl7\" (UID: \"d5aad55c-1b94-460e-96e3-a40d87b421e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-62236a-7fc8b485b4-tspl7"
Oct 30 22:33:14.185661 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:14.185631    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxdwk\" (UniqueName: \"kubernetes.io/projected/d5aad55c-1b94-460e-96e3-a40d87b421e0-kube-api-access-zxdwk\") pod \"maas-api-e2e-api-b-62236a-7fc8b485b4-tspl7\" (UID: \"d5aad55c-1b94-460e-96e3-a40d87b421e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-62236a-7fc8b485b4-tspl7"
Oct 30 22:33:14.670021 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:14.669982    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5aad55c-1b94-460e-96e3-a40d87b421e0-maas-api-tls\") pod \"maas-api-e2e-api-b-62236a-7fc8b485b4-tspl7\" (UID: \"d5aad55c-1b94-460e-96e3-a40d87b421e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-62236a-7fc8b485b4-tspl7"
Oct 30 22:33:14.672391 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:14.672369    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5aad55c-1b94-460e-96e3-a40d87b421e0-maas-api-tls\") pod \"maas-api-e2e-api-b-62236a-7fc8b485b4-tspl7\" (UID: \"d5aad55c-1b94-460e-96e3-a40d87b421e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-62236a-7fc8b485b4-tspl7"
Oct 30 22:33:14.918544 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:14.918504    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-62236a-7fc8b485b4-tspl7"
Oct 30 22:33:15.306966 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:15.306178    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-62236a-7fc8b485b4-tspl7"]
Oct 30 22:33:15.983033 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:15.982958    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-62236a-7fc8b485b4-tspl7" event={"ID":"d5aad55c-1b94-460e-96e3-a40d87b421e0","Type":"ContainerStarted","Data":"b6b7fb24d8968f0aea9b19993483235d19e958e5bd3e0239bde95c454aeb8a55"}
Oct 30 22:33:15.983033 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:15.983008    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-62236a-7fc8b485b4-tspl7" event={"ID":"d5aad55c-1b94-460e-96e3-a40d87b421e0","Type":"ContainerStarted","Data":"4e71befa99319e97f7f873bd4b01061254d99fd5999498aa5e06c58f4d1083f5"}
Oct 30 22:33:15.983246 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:15.983086    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-62236a-7fc8b485b4-tspl7"
Oct 30 22:33:16.025226 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:16.025174    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-62236a-7fc8b485b4-tspl7" podStartSLOduration=2.6382084580000003 podStartE2EDuration="3.025156201s" totalImagesPullingTime="386.947743ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:33:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:33:16.020890889 +0000 UTC m=+1579.946817146" watchObservedRunningTime="2026-10-30 22:33:16.025156201 +0000 UTC m=+1579.951082408"
Oct 30 22:33:19.425950 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:19.425913    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6922d3-544f4f95d4-8psb5"]
Oct 30 22:33:19.431842 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:19.431820    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6922d3-544f4f95d4-8psb5"
Oct 30 22:33:19.454424 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:19.454395    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6922d3-544f4f95d4-8psb5"]
Oct 30 22:33:19.459177 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:19.459135    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-6922d3-serving-cert\""
Oct 30 22:33:19.550564 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:19.550527    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e931bc1-b014-48ed-beaa-e82a64000d6a-maas-api-tls\") pod \"maas-api-e2e-inf-a-6922d3-544f4f95d4-8psb5\" (UID: \"4e931bc1-b014-48ed-beaa-e82a64000d6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6922d3-544f4f95d4-8psb5"
Oct 30 22:33:19.550759 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:19.550730    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2wf6\" (UniqueName: \"kubernetes.io/projected/4e931bc1-b014-48ed-beaa-e82a64000d6a-kube-api-access-j2wf6\") pod \"maas-api-e2e-inf-a-6922d3-544f4f95d4-8psb5\" (UID: \"4e931bc1-b014-48ed-beaa-e82a64000d6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6922d3-544f4f95d4-8psb5"
Oct 30 22:33:19.550876 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:19.550846    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e931bc1-b014-48ed-beaa-e82a64000d6a-metrics-certs\") pod \"maas-api-e2e-inf-a-6922d3-544f4f95d4-8psb5\" (UID: \"4e931bc1-b014-48ed-beaa-e82a64000d6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6922d3-544f4f95d4-8psb5"
Oct 30 22:33:19.551013 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:19.550936    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e931bc1-b014-48ed-beaa-e82a64000d6a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-6922d3-544f4f95d4-8psb5\" (UID: \"4e931bc1-b014-48ed-beaa-e82a64000d6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6922d3-544f4f95d4-8psb5"
Oct 30 22:33:19.652239 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:19.652211    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e931bc1-b014-48ed-beaa-e82a64000d6a-maas-api-tls\") pod \"maas-api-e2e-inf-a-6922d3-544f4f95d4-8psb5\" (UID: \"4e931bc1-b014-48ed-beaa-e82a64000d6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6922d3-544f4f95d4-8psb5"
Oct 30 22:33:19.652376 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:19.652316    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2wf6\" (UniqueName: \"kubernetes.io/projected/4e931bc1-b014-48ed-beaa-e82a64000d6a-kube-api-access-j2wf6\") pod \"maas-api-e2e-inf-a-6922d3-544f4f95d4-8psb5\" (UID: \"4e931bc1-b014-48ed-beaa-e82a64000d6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6922d3-544f4f95d4-8psb5"
Oct 30 22:33:19.652418 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:19.652380    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e931bc1-b014-48ed-beaa-e82a64000d6a-metrics-certs\") pod \"maas-api-e2e-inf-a-6922d3-544f4f95d4-8psb5\" (UID: \"4e931bc1-b014-48ed-beaa-e82a64000d6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6922d3-544f4f95d4-8psb5"
Oct 30 22:33:19.652455 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:19.652416    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e931bc1-b014-48ed-beaa-e82a64000d6a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-6922d3-544f4f95d4-8psb5\" (UID: \"4e931bc1-b014-48ed-beaa-e82a64000d6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6922d3-544f4f95d4-8psb5"
Oct 30 22:33:19.653102 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:19.653066    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e931bc1-b014-48ed-beaa-e82a64000d6a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-6922d3-544f4f95d4-8psb5\" (UID: \"4e931bc1-b014-48ed-beaa-e82a64000d6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6922d3-544f4f95d4-8psb5"
Oct 30 22:33:19.654824 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:19.654794    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e931bc1-b014-48ed-beaa-e82a64000d6a-metrics-certs\") pod \"maas-api-e2e-inf-a-6922d3-544f4f95d4-8psb5\" (UID: \"4e931bc1-b014-48ed-beaa-e82a64000d6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6922d3-544f4f95d4-8psb5"
Oct 30 22:33:19.654925 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:19.654886    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e931bc1-b014-48ed-beaa-e82a64000d6a-maas-api-tls\") pod \"maas-api-e2e-inf-a-6922d3-544f4f95d4-8psb5\" (UID: \"4e931bc1-b014-48ed-beaa-e82a64000d6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6922d3-544f4f95d4-8psb5"
Oct 30 22:33:19.669774 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:19.669754    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2wf6\" (UniqueName: \"kubernetes.io/projected/4e931bc1-b014-48ed-beaa-e82a64000d6a-kube-api-access-j2wf6\") pod \"maas-api-e2e-inf-a-6922d3-544f4f95d4-8psb5\" (UID: \"4e931bc1-b014-48ed-beaa-e82a64000d6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6922d3-544f4f95d4-8psb5"
Oct 30 22:33:19.762864 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:19.762809    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6922d3-544f4f95d4-8psb5"
Oct 30 22:33:20.071462 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:20.071419    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6922d3-544f4f95d4-8psb5"]
Oct 30 22:33:20.073308 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:33:20.073270    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e931bc1_b014_48ed_beaa_e82a64000d6a.slice/crio-5d17fdcd82691b83abe1909fd536e22c6388addec35045756f4357ebcbe3870d WatchSource:0}: Error finding container 5d17fdcd82691b83abe1909fd536e22c6388addec35045756f4357ebcbe3870d: Status 404 returned error can't find the container with id 5d17fdcd82691b83abe1909fd536e22c6388addec35045756f4357ebcbe3870d
Oct 30 22:33:21.016657 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:21.016607    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6922d3-544f4f95d4-8psb5" event={"ID":"4e931bc1-b014-48ed-beaa-e82a64000d6a","Type":"ContainerStarted","Data":"d6fb0bb9b68e9775456b332e8dfc3789151e4d33e1f4bed1231e59c92a8a9255"}
Oct 30 22:33:21.017127 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:21.016677    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6922d3-544f4f95d4-8psb5" event={"ID":"4e931bc1-b014-48ed-beaa-e82a64000d6a","Type":"ContainerStarted","Data":"5d17fdcd82691b83abe1909fd536e22c6388addec35045756f4357ebcbe3870d"}
Oct 30 22:33:21.017127 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:21.016707    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6922d3-544f4f95d4-8psb5"
Oct 30 22:33:21.056477 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:21.056428    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6922d3-544f4f95d4-8psb5" podStartSLOduration=1.75881855 podStartE2EDuration="2.056415103s" totalImagesPullingTime="297.596553ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:33:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:33:21.053404513 +0000 UTC m=+1584.979330719" watchObservedRunningTime="2026-10-30 22:33:21.056415103 +0000 UTC m=+1584.982341335"
Oct 30 22:33:21.994810 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:21.994780    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-62236a-7fc8b485b4-tspl7"
Oct 30 22:33:22.318475 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:22.318357    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6bz92"]
Oct 30 22:33:26.267097 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:33:26.267027    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 22:33:27.029157 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:27.029125    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6922d3-544f4f95d4-8psb5"
Oct 30 22:33:42.991985 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:42.991947    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-4cm9t"]
Oct 30 22:33:42.992460 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:42.992270    4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-4cm9t" podUID="1c0da8b6-d70a-44bc-81cc-8e79df362b32" containerName="istio-proxy" containerID="cri-o://a3cbda41180568f1eb03879a3c61938fc6a04eba33291b908909ff6dbef6b791" gracePeriod=30
Oct 30 22:33:43.006641 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:43.006615    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-9nsmf"]
Oct 30 22:33:43.006857 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:43.006833    4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-9nsmf" podUID="abdba985-1e18-4dc3-b3bb-2456a71609f2" containerName="istio-proxy" containerID="cri-o://824f67ed98229f197007d8bddfb896de48cc707fd203edcda71ab8f591ee486f" gracePeriod=30
Oct 30 22:33:43.010250 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:43.010221    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-2bzhm"]
Oct 30 22:33:43.010672 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:43.010603    4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-2bzhm" podUID="fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92" containerName="istio-proxy" containerID="cri-o://08dd29e2e4f609d04f560a5688d0ea2a2aaf0f43b6aac4dcd86b8129c0ef0c6f" gracePeriod=30
Oct 30 22:33:47.902353 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:47.902320    4477 patch_prober.go:29] interesting pod/e2e-api-a-370ec2-openshift-default-6fccc4f697-9nsmf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:33:47.903152 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:47.902384    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-9nsmf" podUID="abdba985-1e18-4dc3-b3bb-2456a71609f2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:33:48.008459 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:48.008422    4477 patch_prober.go:29] interesting pod/e2e-api-a-370ec2-openshift-default-6fccc4f697-4cm9t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.102:15021/healthz/ready\": read tcp 10.134.0.2:34298->10.134.0.102:15021: read: connection reset by peer" start-of-body=
Oct 30 22:33:48.008638 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:48.008494    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-4cm9t" podUID="1c0da8b6-d70a-44bc-81cc-8e79df362b32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.102:15021/healthz/ready\": read tcp 10.134.0.2:34298->10.134.0.102:15021: read: connection reset by peer"
Oct 30 22:33:48.126766 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:48.126738    4477 patch_prober.go:29] interesting pod/e2e-api-a-370ec2-openshift-default-6fccc4f697-2bzhm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body=
Oct 30 22:33:48.126868 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:48.126798    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-2bzhm" podUID="fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused"
Oct 30 22:33:48.238785 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:48.238743    4477 generic.go:356] "Generic (PLEG): container finished" podID="abdba985-1e18-4dc3-b3bb-2456a71609f2" containerID="824f67ed98229f197007d8bddfb896de48cc707fd203edcda71ab8f591ee486f" exitCode=0
Oct 30 22:33:48.238926 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:48.238872    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-9nsmf" event={"ID":"abdba985-1e18-4dc3-b3bb-2456a71609f2","Type":"ContainerDied","Data":"824f67ed98229f197007d8bddfb896de48cc707fd203edcda71ab8f591ee486f"}
Oct 30 22:33:48.240731 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:48.240699    4477 generic.go:356] "Generic (PLEG): container finished" podID="fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92" containerID="08dd29e2e4f609d04f560a5688d0ea2a2aaf0f43b6aac4dcd86b8129c0ef0c6f" exitCode=0
Oct 30 22:33:48.240895 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:48.240879    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-2bzhm" event={"ID":"fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92","Type":"ContainerDied","Data":"08dd29e2e4f609d04f560a5688d0ea2a2aaf0f43b6aac4dcd86b8129c0ef0c6f"}
Oct 30 22:33:48.242743 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:48.242716    4477 generic.go:356] "Generic (PLEG): container finished" podID="1c0da8b6-d70a-44bc-81cc-8e79df362b32" containerID="a3cbda41180568f1eb03879a3c61938fc6a04eba33291b908909ff6dbef6b791" exitCode=0
Oct 30 22:33:48.242836 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:48.242811    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-4cm9t" event={"ID":"1c0da8b6-d70a-44bc-81cc-8e79df362b32","Type":"ContainerDied","Data":"a3cbda41180568f1eb03879a3c61938fc6a04eba33291b908909ff6dbef6b791"}
Oct 30 22:33:48.379903 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:48.379870    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-4cm9t"
Oct 30 22:33:48.410222 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:48.410191    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-9nsmf"
Oct 30 22:33:48.415323 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:48.415294    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-2bzhm"
Oct 30 22:33:48.422480 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:48.422449    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c0da8b6-d70a-44bc-81cc-8e79df362b32-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c0da8b6-d70a-44bc-81cc-8e79df362b32-workload-socket\") pod \"1c0da8b6-d70a-44bc-81cc-8e79df362b32\" (UID: \"1c0da8b6-d70a-44bc-81cc-8e79df362b32\") "
Oct 30 22:33:48.422634 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:48.422529    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c0da8b6-d70a-44bc-81cc-8e79df362b32-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c0da8b6-d70a-44bc-81cc-8e79df362b32-istio-data\") pod \"1c0da8b6-d70a-44bc-81cc-8e79df362b32\" (UID: \"1c0da8b6-d70a-44bc-81cc-8e79df362b32\") "
Oct 30 22:33:48.422634 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:48.422559    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c0da8b6-d70a-44bc-81cc-8e79df362b32-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c0da8b6-d70a-44bc-81cc-8e79df362b32-credential-socket\") pod \"1c0da8b6-d70a-44bc-81cc-8e79df362b32\" (UID: \"1c0da8b6-d70a-44bc-81cc-8e79df362b32\") "
Oct 30 22:33:48.422634 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:48.422600    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1c0da8b6-d70a-44bc-81cc-8e79df362b32-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c0da8b6-d70a-44bc-81cc-8e79df362b32-istio-podinfo\") pod \"1c0da8b6-d70a-44bc-81cc-8e79df362b32\" (UID: \"1c0da8b6-d70a-44bc-81cc-8e79df362b32\") "
Oct 30 22:33:48.422814 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:48.422655    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c0da8b6-d70a-44bc-81cc-8e79df362b32-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c0da8b6-d70a-44bc-81cc-8e79df362b32-istio-envoy\") pod \"1c0da8b6-d70a-44bc-81cc-8e79df362b32\" (UID: \"1c0da8b6-d70a-44bc-81cc-8e79df362b32\") "
Oct 30 22:33:48.422814 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:48.422777    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c0da8b6-d70a-44bc-81cc-8e79df362b32-kube-api-access-nh8tl\" (UniqueName: \"kubernetes.io/projected/1c0da8b6-d70a-44bc-81cc-8e79df362b32-kube-api-access-nh8tl\") pod \"1c0da8b6-d70a-44bc-81cc-8e79df362b32\" (UID: \"1c0da8b6-d70a-44bc-81cc-8e79df362b32\") "
Oct 30 22:33:48.422923 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:48.422821    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c0da8b6-d70a-44bc-81cc-8e79df362b32-workload-socket" pod "1c0da8b6-d70a-44bc-81cc-8e79df362b32" (UID: "1c0da8b6-d70a-44bc-81cc-8e79df362b32"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:33:48.422923 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:48.422857    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c0da8b6-d70a-44bc-81cc-8e79df362b32-istio-token\" (UniqueName: \"kubernetes.io/projected/1c0da8b6-d70a-44bc-81cc-8e79df362b32-istio-token\") pod \"1c0da8b6-d70a-44bc-81cc-8e79df362b32\" (UID: \"1c0da8b6-d70a-44bc-81cc-8e79df362b32\") "
Oct 30 22:33:48.423036 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:48.422944    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c0da8b6-d70a-44bc-81cc-8e79df362b32-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c0da8b6-d70a-44bc-81cc-8e79df362b32-istiod-ca-cert\") pod \"1c0da8b6-d70a-44bc-81cc-8e79df362b32\" (UID: \"1c0da8b6-d70a-44bc-81cc-8e79df362b32\") "
Oct 30 22:33:48.423036 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:48.422979    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c0da8b6-d70a-44bc-81cc-8e79df362b32-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c0da8b6-d70a-44bc-81cc-8e79df362b32-workload-certs\") pod \"1c0da8b6-d70a-44bc-81cc-8e79df362b32\" (UID: \"1c0da8b6-d70a-44bc-81cc-8e79df362b32\") "
Oct 30 22:33:48.423036 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:48.422996    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c0da8b6-d70a-44bc-81cc-8e79df362b32-credential-socket" pod "1c0da8b6-d70a-44bc-81cc-8e79df362b32" (UID: "1c0da8b6-d70a-44bc-81cc-8e79df362b32"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:33:48.423589 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:48.423561    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c0da8b6-d70a-44bc-81cc-8e79df362b32-workload-certs" pod "1c0da8b6-d70a-44bc-81cc-8e79df362b32" (UID: "1c0da8b6-d70a-44bc-81cc-8e79df362b32"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:33:48.423707 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:48.423659    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c0da8b6-d70a-44bc-81cc-8e79df362b32-istiod-ca-cert" pod "1c0da8b6-d70a-44bc-81cc-8e79df362b32" (UID: "1c0da8b6-d70a-44bc-81cc-8e79df362b32"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:33:48.423822 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:48.423802    4477 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c0da8b6-d70a-44bc-81cc-8e79df362b32-istiod-ca-cert\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:33:48.423882 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:48.423826    4477 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c0da8b6-d70a-44bc-81cc-8e79df362b32-workload-certs\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:33:48.423882 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:48.423840    4477 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c0da8b6-d70a-44bc-81cc-8e79df362b32-workload-socket\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:33:48.423991 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:48.423941    4477 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c0da8b6-d70a-44bc-81cc-8e79df362b32-credential-socket\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:33:48.423991 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:48.423950    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c0da8b6-d70a-44bc-81cc-8e79df362b32-istio-data" pod "1c0da8b6-d70a-44bc-81cc-8e79df362b32" (UID: "1c0da8b6-d70a-44bc-81cc-8e79df362b32"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:33:48.427093 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:48.426881    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c0da8b6-d70a-44bc-81cc-8e79df362b32-istio-envoy" pod "1c0da8b6-d70a-44bc-81cc-8e79df362b32" (UID: "1c0da8b6-d70a-44bc-81cc-8e79df362b32"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:33:48.427212 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:48.427079    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1c0da8b6-d70a-44bc-81cc-8e79df362b32-istio-podinfo" pod "1c0da8b6-d70a-44bc-81cc-8e79df362b32" (UID: "1c0da8b6-d70a-44bc-81cc-8e79df362b32"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:33:48.427512 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:48.427490    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c0da8b6-d70a-44bc-81cc-8e79df362b32-kube-api-access-nh8tl" pod "1c0da8b6-d70a-44bc-81cc-8e79df362b32" (UID: "1c0da8b6-d70a-44bc-81cc-8e79df362b32"). InnerVolumeSpecName "kube-api-access-nh8tl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:33:48.429497 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:48.429472    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c0da8b6-d70a-44bc-81cc-8e79df362b32-istio-token" pod "1c0da8b6-d70a-44bc-81cc-8e79df362b32" (UID: "1c0da8b6-d70a-44bc-81cc-8e79df362b32"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:33:48.524602 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:48.524500    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abdba985-1e18-4dc3-b3bb-2456a71609f2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abdba985-1e18-4dc3-b3bb-2456a71609f2-workload-socket\") pod \"abdba985-1e18-4dc3-b3bb-2456a71609f2\" (UID: \"abdba985-1e18-4dc3-b3bb-2456a71609f2\") "
Oct 30 22:33:48.524602 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:48.524553    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abdba985-1e18-4dc3-b3bb-2456a71609f2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abdba985-1e18-4dc3-b3bb-2456a71609f2-credential-socket\") pod \"abdba985-1e18-4dc3-b3bb-2456a71609f2\" (UID: \"abdba985-1e18-4dc3-b3bb-2456a71609f2\") "
Oct 30 22:33:48.524800 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:48.524613    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92-istio-data\") pod \"fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92\" (UID: \"fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92\") "
Oct 30 22:33:48.524800 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:48.524646    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92-istio-podinfo\") pod \"fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92\" (UID: \"fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92\") "
Oct 30 22:33:48.524800 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:48.524739    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abdba985-1e18-4dc3-b3bb-2456a71609f2-kube-api-access-k7nqj\" (UniqueName: \"kubernetes.io/projected/abdba985-1e18-4dc3-b3bb-2456a71609f2-kube-api-access-k7nqj\") pod \"abdba985-1e18-4dc3-b3bb-2456a71609f2\" (UID: \"abdba985-1e18-4dc3-b3bb-2456a71609f2\") "
Oct 30 22:33:48.524981 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:48.524938    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abdba985-1e18-4dc3-b3bb-2456a71609f2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abdba985-1e18-4dc3-b3bb-2456a71609f2-workload-certs\") pod \"abdba985-1e18-4dc3-b3bb-2456a71609f2\" (UID: \"abdba985-1e18-4dc3-b3bb-2456a71609f2\") "
Oct 30 22:33:48.525101 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:48.525082    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92-istiod-ca-cert\") pod \"fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92\" (UID: \"fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92\") "
Oct 30 22:33:48.525192 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:48.525119    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92-kube-api-access-nc85x\" (UniqueName: \"kubernetes.io/projected/fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92-kube-api-access-nc85x\") pod \"fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92\" (UID: \"fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92\") "
Oct 30 22:33:48.525282 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:48.525192    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92-credential-socket\") pod \"fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92\" (UID: \"fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92\") "
Oct 30 22:33:48.525282 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:48.525248    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92-istio-token\" (UniqueName: \"kubernetes.io/projected/fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92-istio-token\") pod \"fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92\" (UID: \"fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92\") "
Oct 30 22:33:48.525389 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:48.525304    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92-workload-certs\") pod \"fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92\" (UID: \"fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92\") "
Oct 30 22:33:48.525447 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:48.525407    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abdba985-1e18-4dc3-b3bb-2456a71609f2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abdba985-1e18-4dc3-b3bb-2456a71609f2-istio-data\") pod \"abdba985-1e18-4dc3-b3bb-2456a71609f2\" (UID: \"abdba985-1e18-4dc3-b3bb-2456a71609f2\") "
Oct 30 22:33:48.525447 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:48.525439    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92-workload-socket\") pod \"fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92\" (UID: \"fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92\") "
Oct 30 22:33:48.525546 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:48.525479    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abdba985-1e18-4dc3-b3bb-2456a71609f2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abdba985-1e18-4dc3-b3bb-2456a71609f2-istiod-ca-cert\") pod \"abdba985-1e18-4dc3-b3bb-2456a71609f2\" (UID: \"abdba985-1e18-4dc3-b3bb-2456a71609f2\") "
Oct 30 22:33:48.525546 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:48.525502    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abdba985-1e18-4dc3-b3bb-2456a71609f2-istio-token\" (UniqueName: \"kubernetes.io/projected/abdba985-1e18-4dc3-b3bb-2456a71609f2-istio-token\") pod \"abdba985-1e18-4dc3-b3bb-2456a71609f2\" (UID: \"abdba985-1e18-4dc3-b3bb-2456a71609f2\") "
Oct 30 22:33:48.525546 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:48.525522    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92-credential-socket" pod "fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92" (UID: "fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:33:48.525546 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:48.525534    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/abdba985-1e18-4dc3-b3bb-2456a71609f2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abdba985-1e18-4dc3-b3bb-2456a71609f2-istio-podinfo\") pod \"abdba985-1e18-4dc3-b3bb-2456a71609f2\" (UID: \"abdba985-1e18-4dc3-b3bb-2456a71609f2\") "
Oct 30 22:33:48.525762 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:48.525529    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92-istiod-ca-cert" pod "fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92" (UID: "fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:33:48.525762 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:48.525561    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92-istio-envoy\") pod \"fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92\" (UID: \"fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92\") "
Oct 30 22:33:48.525762 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:48.525667    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abdba985-1e18-4dc3-b3bb-2456a71609f2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abdba985-1e18-4dc3-b3bb-2456a71609f2-istio-envoy\") pod \"abdba985-1e18-4dc3-b3bb-2456a71609f2\" (UID: \"abdba985-1e18-4dc3-b3bb-2456a71609f2\") "
Oct 30 22:33:48.525979 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:48.525955    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92-workload-certs" pod "fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92" (UID: "fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:33:48.526412 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:48.526072    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92-istio-data" pod "fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92" (UID: "fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:33:48.526412 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:48.526118    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92-workload-socket" pod "fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92" (UID: "fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:33:48.526412 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:48.526286    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abdba985-1e18-4dc3-b3bb-2456a71609f2-workload-socket" pod "abdba985-1e18-4dc3-b3bb-2456a71609f2" (UID: "abdba985-1e18-4dc3-b3bb-2456a71609f2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:33:48.526666 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:48.526421    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abdba985-1e18-4dc3-b3bb-2456a71609f2-workload-certs" pod "abdba985-1e18-4dc3-b3bb-2456a71609f2" (UID: "abdba985-1e18-4dc3-b3bb-2456a71609f2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:33:48.526666 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:48.526621    4477 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c0da8b6-d70a-44bc-81cc-8e79df362b32-istio-token\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:33:48.526666 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:48.526640    4477 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abdba985-1e18-4dc3-b3bb-2456a71609f2-workload-certs\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:33:48.526666 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:48.526652    4477 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92-istiod-ca-cert\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:33:48.526666 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:48.526665    4477 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92-credential-socket\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:33:48.526926 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:48.526678    4477 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92-workload-certs\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:33:48.526926 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:48.526693    4477 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c0da8b6-d70a-44bc-81cc-8e79df362b32-istio-data\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:33:48.526926 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:48.526713    4477 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c0da8b6-d70a-44bc-81cc-8e79df362b32-istio-podinfo\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:33:48.526926 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:48.526724    4477 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92-workload-socket\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:33:48.526926 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:48.526739    4477 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c0da8b6-d70a-44bc-81cc-8e79df362b32-istio-envoy\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:33:48.526926 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:48.526752    4477 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abdba985-1e18-4dc3-b3bb-2456a71609f2-workload-socket\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:33:48.526926 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:48.526765    4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nh8tl\" (UniqueName: \"kubernetes.io/projected/1c0da8b6-d70a-44bc-81cc-8e79df362b32-kube-api-access-nh8tl\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:33:48.526926 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:48.526778    4477 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92-istio-data\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:33:48.526926 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:48.526827    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abdba985-1e18-4dc3-b3bb-2456a71609f2-credential-socket" pod "abdba985-1e18-4dc3-b3bb-2456a71609f2" (UID: "abdba985-1e18-4dc3-b3bb-2456a71609f2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:33:48.526926 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:48.526898    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abdba985-1e18-4dc3-b3bb-2456a71609f2-istio-data" pod "abdba985-1e18-4dc3-b3bb-2456a71609f2" (UID: "abdba985-1e18-4dc3-b3bb-2456a71609f2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:33:48.527684 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:48.527414    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abdba985-1e18-4dc3-b3bb-2456a71609f2-istiod-ca-cert" pod "abdba985-1e18-4dc3-b3bb-2456a71609f2" (UID: "abdba985-1e18-4dc3-b3bb-2456a71609f2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:33:48.528103 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:48.528035    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92-istio-podinfo" pod "fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92" (UID: "fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:33:48.529322 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:48.529293    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92-kube-api-access-nc85x" pod "fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92" (UID: "fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92"). InnerVolumeSpecName "kube-api-access-nc85x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:33:48.530130 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:48.530108    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abdba985-1e18-4dc3-b3bb-2456a71609f2-kube-api-access-k7nqj" pod "abdba985-1e18-4dc3-b3bb-2456a71609f2" (UID: "abdba985-1e18-4dc3-b3bb-2456a71609f2"). InnerVolumeSpecName "kube-api-access-k7nqj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:33:48.531384 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:48.531360    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/abdba985-1e18-4dc3-b3bb-2456a71609f2-istio-podinfo" pod "abdba985-1e18-4dc3-b3bb-2456a71609f2" (UID: "abdba985-1e18-4dc3-b3bb-2456a71609f2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:33:48.536372 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:48.536349    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abdba985-1e18-4dc3-b3bb-2456a71609f2-istio-token" pod "abdba985-1e18-4dc3-b3bb-2456a71609f2" (UID: "abdba985-1e18-4dc3-b3bb-2456a71609f2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:33:48.536454 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:48.536366    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92-istio-envoy" pod "fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92" (UID: "fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:33:48.536454 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:48.536388    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92-istio-token" pod "fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92" (UID: "fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:33:48.537640 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:48.537622    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abdba985-1e18-4dc3-b3bb-2456a71609f2-istio-envoy" pod "abdba985-1e18-4dc3-b3bb-2456a71609f2" (UID: "abdba985-1e18-4dc3-b3bb-2456a71609f2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:33:48.628661 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:48.628630    4477 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92-istio-token\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:33:48.628661 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:48.628656    4477 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abdba985-1e18-4dc3-b3bb-2456a71609f2-istio-data\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:33:48.628661 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:48.628667    4477 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abdba985-1e18-4dc3-b3bb-2456a71609f2-istiod-ca-cert\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:33:48.628861 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:48.628675    4477 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abdba985-1e18-4dc3-b3bb-2456a71609f2-istio-token\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:33:48.628861 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:48.628684    4477 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abdba985-1e18-4dc3-b3bb-2456a71609f2-istio-podinfo\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:33:48.628861 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:48.628691    4477 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92-istio-envoy\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:33:48.628861 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:48.628699    4477 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abdba985-1e18-4dc3-b3bb-2456a71609f2-istio-envoy\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:33:48.628861 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:48.628707    4477 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abdba985-1e18-4dc3-b3bb-2456a71609f2-credential-socket\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:33:48.628861 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:48.628716    4477 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92-istio-podinfo\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:33:48.628861 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:48.628725    4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7nqj\" (UniqueName: \"kubernetes.io/projected/abdba985-1e18-4dc3-b3bb-2456a71609f2-kube-api-access-k7nqj\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:33:48.628861 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:48.628734    4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nc85x\" (UniqueName: \"kubernetes.io/projected/fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92-kube-api-access-nc85x\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:33:49.250708 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:49.250672    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-9nsmf"
Oct 30 22:33:49.250708 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:49.250694    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-9nsmf" event={"ID":"abdba985-1e18-4dc3-b3bb-2456a71609f2","Type":"ContainerDied","Data":"0a515fed01402c1da4b69b1501e10f64ce488c8318e4210d4404ed74409de319"}
Oct 30 22:33:49.251318 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:49.250761    4477 scope.go:122] "RemoveContainer" containerID="824f67ed98229f197007d8bddfb896de48cc707fd203edcda71ab8f591ee486f"
Oct 30 22:33:49.252232 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:49.252214    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-2bzhm"
Oct 30 22:33:49.252307 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:49.252257    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-2bzhm" event={"ID":"fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92","Type":"ContainerDied","Data":"a0f080387c73f40a1d4c1660da44ea045deb43379a835c0dff38ce992af67f69"}
Oct 30 22:33:49.255735 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:49.255712    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-4cm9t"
Oct 30 22:33:49.255735 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:49.255728    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-4cm9t" event={"ID":"1c0da8b6-d70a-44bc-81cc-8e79df362b32","Type":"ContainerDied","Data":"186c3023acb3e3656b222886fb5f236668889e7bc187306e932d3a16dfd70f19"}
Oct 30 22:33:49.264251 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:49.264236    4477 scope.go:122] "RemoveContainer" containerID="08dd29e2e4f609d04f560a5688d0ea2a2aaf0f43b6aac4dcd86b8129c0ef0c6f"
Oct 30 22:33:49.274781 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:49.274766    4477 scope.go:122] "RemoveContainer" containerID="a3cbda41180568f1eb03879a3c61938fc6a04eba33291b908909ff6dbef6b791"
Oct 30 22:33:49.308623 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:49.308602    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-9nsmf"]
Oct 30 22:33:49.318902 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:49.318881    4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-9nsmf"]
Oct 30 22:33:49.349346 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:49.349325    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-4cm9t"]
Oct 30 22:33:49.353405 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:49.353386    4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-4cm9t"]
Oct 30 22:33:49.393576 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:49.393547    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-2bzhm"]
Oct 30 22:33:49.393750 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:49.393597    4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-370ec2-openshift-default-6fccc4f697-2bzhm"]
Oct 30 22:33:50.194984 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:50.194951    4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c0da8b6-d70a-44bc-81cc-8e79df362b32" path="/var/lib/kubelet/pods/1c0da8b6-d70a-44bc-81cc-8e79df362b32/volumes"
Oct 30 22:33:50.195598 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:50.195580    4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abdba985-1e18-4dc3-b3bb-2456a71609f2" path="/var/lib/kubelet/pods/abdba985-1e18-4dc3-b3bb-2456a71609f2/volumes"
Oct 30 22:33:50.196069 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:33:50.196056    4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92" path="/var/lib/kubelet/pods/fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92/volumes"
Oct 30 22:33:56.268382 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:33:56.268340    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 22:34:14.772188 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:34:14.772155    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-7bhqj"]
Oct 30 22:34:14.772928 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:34:14.772909    4477 cpu_manager.go:414] "containerMap: removing container" podUID="abdba985-1e18-4dc3-b3bb-2456a71609f2" containerName="istio-proxy"
Oct 30 22:34:14.772928 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:34:14.772928    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abdba985-1e18-4dc3-b3bb-2456a71609f2" containerName="istio-proxy"
Oct 30 22:34:14.773081 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:34:14.772946    4477 cpu_manager.go:414] "containerMap: removing container" podUID="fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92" containerName="istio-proxy"
Oct 30 22:34:14.773081 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:34:14.772952    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92" containerName="istio-proxy"
Oct 30 22:34:14.773081 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:34:14.772988    4477 cpu_manager.go:414] "containerMap: removing container" podUID="1c0da8b6-d70a-44bc-81cc-8e79df362b32" containerName="istio-proxy"
Oct 30 22:34:14.773081 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:34:14.772994    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c0da8b6-d70a-44bc-81cc-8e79df362b32" containerName="istio-proxy"
Oct 30 22:34:14.773239 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:34:14.773105    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c0da8b6-d70a-44bc-81cc-8e79df362b32" containerName="istio-proxy"
Oct 30 22:34:14.773239 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:34:14.773119    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="abdba985-1e18-4dc3-b3bb-2456a71609f2" containerName="istio-proxy"
Oct 30 22:34:14.773239 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:34:14.773132    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe4a6d2a-5ccd-400d-9948-c7da6ffa4f92" containerName="istio-proxy"
Oct 30 22:34:14.779059 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:34:14.779015    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-7bhqj"
Oct 30 22:34:14.812521 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:34:14.812489    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-7bhqj"]
Oct 30 22:34:14.881782 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:34:14.881755    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-6922d3-openshift-default-dockercfg-5ldsg\""
Oct 30 22:34:14.896589 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:34:14.896561    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58693405-8a5b-4841-865d-aa29cc320e4d-istiod-ca-cert\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-7bhqj\" (UID: \"58693405-8a5b-4841-865d-aa29cc320e4d\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-7bhqj"
Oct 30 22:34:14.896727 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:34:14.896603    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58693405-8a5b-4841-865d-aa29cc320e4d-istio-podinfo\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-7bhqj\" (UID: \"58693405-8a5b-4841-865d-aa29cc320e4d\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-7bhqj"
Oct 30 22:34:14.896727 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:34:14.896686    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slmtv\" (UniqueName: \"kubernetes.io/projected/58693405-8a5b-4841-865d-aa29cc320e4d-kube-api-access-slmtv\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-7bhqj\" (UID: \"58693405-8a5b-4841-865d-aa29cc320e4d\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-7bhqj"
Oct 30 22:34:14.896727 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:34:14.896721    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58693405-8a5b-4841-865d-aa29cc320e4d-credential-socket\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-7bhqj\" (UID: \"58693405-8a5b-4841-865d-aa29cc320e4d\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-7bhqj"
Oct 30 22:34:14.896855 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:34:14.896804    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58693405-8a5b-4841-865d-aa29cc320e4d-istio-data\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-7bhqj\" (UID: \"58693405-8a5b-4841-865d-aa29cc320e4d\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-7bhqj"
Oct 30 22:34:14.896896 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:34:14.896857    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58693405-8a5b-4841-865d-aa29cc320e4d-workload-certs\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-7bhqj\" (UID: \"58693405-8a5b-4841-865d-aa29cc320e4d\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-7bhqj"
Oct 30 22:34:14.896896 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:34:14.896887    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58693405-8a5b-4841-865d-aa29cc320e4d-istio-token\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-7bhqj\" (UID: \"58693405-8a5b-4841-865d-aa29cc320e4d\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-7bhqj"
Oct 30 22:34:14.897030 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:34:14.897000    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58693405-8a5b-4841-865d-aa29cc320e4d-workload-socket\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-7bhqj\" (UID: \"58693405-8a5b-4841-865d-aa29cc320e4d\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-7bhqj"
Oct 30 22:34:14.897168 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:34:14.897131    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58693405-8a5b-4841-865d-aa29cc320e4d-istio-envoy\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-7bhqj\" (UID: \"58693405-8a5b-4841-865d-aa29cc320e4d\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-7bhqj"
Oct 30 22:34:14.998591 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:34:14.998567    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58693405-8a5b-4841-865d-aa29cc320e4d-istio-envoy\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-7bhqj\" (UID: \"58693405-8a5b-4841-865d-aa29cc320e4d\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-7bhqj"
Oct 30 22:34:14.998724 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:34:14.998675    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58693405-8a5b-4841-865d-aa29cc320e4d-istiod-ca-cert\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-7bhqj\" (UID: \"58693405-8a5b-4841-865d-aa29cc320e4d\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-7bhqj"
Oct 30 22:34:14.998724 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:34:14.998700    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58693405-8a5b-4841-865d-aa29cc320e4d-istio-podinfo\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-7bhqj\" (UID: \"58693405-8a5b-4841-865d-aa29cc320e4d\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-7bhqj"
Oct 30 22:34:14.998724 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:34:14.998717    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slmtv\" (UniqueName: \"kubernetes.io/projected/58693405-8a5b-4841-865d-aa29cc320e4d-kube-api-access-slmtv\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-7bhqj\" (UID: \"58693405-8a5b-4841-865d-aa29cc320e4d\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-7bhqj"
Oct 30 22:34:14.998859 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:34:14.998736    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58693405-8a5b-4841-865d-aa29cc320e4d-credential-socket\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-7bhqj\" (UID: \"58693405-8a5b-4841-865d-aa29cc320e4d\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-7bhqj"
Oct 30 22:34:14.998859 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:34:14.998788    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58693405-8a5b-4841-865d-aa29cc320e4d-istio-data\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-7bhqj\" (UID: \"58693405-8a5b-4841-865d-aa29cc320e4d\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-7bhqj"
Oct 30 22:34:14.998859 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:34:14.998829    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58693405-8a5b-4841-865d-aa29cc320e4d-workload-certs\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-7bhqj\" (UID: \"58693405-8a5b-4841-865d-aa29cc320e4d\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-7bhqj"
Oct 30 22:34:14.999014 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:34:14.998867    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58693405-8a5b-4841-865d-aa29cc320e4d-istio-token\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-7bhqj\" (UID: \"58693405-8a5b-4841-865d-aa29cc320e4d\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-7bhqj"
Oct 30 22:34:14.999014 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:34:14.998917    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58693405-8a5b-4841-865d-aa29cc320e4d-workload-socket\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-7bhqj\" (UID: \"58693405-8a5b-4841-865d-aa29cc320e4d\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-7bhqj"
Oct 30 22:34:14.999300 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:34:14.999246    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58693405-8a5b-4841-865d-aa29cc320e4d-istio-data\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-7bhqj\" (UID: \"58693405-8a5b-4841-865d-aa29cc320e4d\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-7bhqj"
Oct 30 22:34:14.999406 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:34:14.999327    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58693405-8a5b-4841-865d-aa29cc320e4d-credential-socket\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-7bhqj\" (UID: \"58693405-8a5b-4841-865d-aa29cc320e4d\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-7bhqj"
Oct 30 22:34:14.999406 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:34:14.999367    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58693405-8a5b-4841-865d-aa29cc320e4d-workload-certs\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-7bhqj\" (UID: \"58693405-8a5b-4841-865d-aa29cc320e4d\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-7bhqj"
Oct 30 22:34:14.999406 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:34:14.999386    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58693405-8a5b-4841-865d-aa29cc320e4d-workload-socket\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-7bhqj\" (UID: \"58693405-8a5b-4841-865d-aa29cc320e4d\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-7bhqj"
Oct 30 22:34:14.999587 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:34:14.999493    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58693405-8a5b-4841-865d-aa29cc320e4d-istiod-ca-cert\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-7bhqj\" (UID: \"58693405-8a5b-4841-865d-aa29cc320e4d\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-7bhqj"
Oct 30 22:34:15.000948 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:34:15.000928    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58693405-8a5b-4841-865d-aa29cc320e4d-istio-envoy\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-7bhqj\" (UID: \"58693405-8a5b-4841-865d-aa29cc320e4d\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-7bhqj"
Oct 30 22:34:15.010026 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:34:15.010001    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58693405-8a5b-4841-865d-aa29cc320e4d-istio-podinfo\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-7bhqj\" (UID: \"58693405-8a5b-4841-865d-aa29cc320e4d\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-7bhqj"
Oct 30 22:34:15.019191 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:34:15.019159    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slmtv\" (UniqueName: \"kubernetes.io/projected/58693405-8a5b-4841-865d-aa29cc320e4d-kube-api-access-slmtv\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-7bhqj\" (UID: \"58693405-8a5b-4841-865d-aa29cc320e4d\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-7bhqj"
Oct 30 22:34:15.021001 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:34:15.020984    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58693405-8a5b-4841-865d-aa29cc320e4d-istio-token\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-7bhqj\" (UID: \"58693405-8a5b-4841-865d-aa29cc320e4d\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-7bhqj"
Oct 30 22:34:15.094318 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:34:15.094253    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-7bhqj"
Oct 30 22:34:15.415142 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:34:15.414290    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-7bhqj"]
Oct 30 22:34:15.417739 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:34:15.417709    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58693405_8a5b_4841_865d_aa29cc320e4d.slice/crio-1edf19ed8cd3c4cd4b9e246a49dbc154833867571d8ddf7025e5834697bc91bf WatchSource:0}: Error finding container 1edf19ed8cd3c4cd4b9e246a49dbc154833867571d8ddf7025e5834697bc91bf: Status 404 returned error can't find the container with id 1edf19ed8cd3c4cd4b9e246a49dbc154833867571d8ddf7025e5834697bc91bf
Oct 30 22:34:15.420833 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:34:15.420797    4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:34:15.420982 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:34:15.420881    4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:34:15.420982 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:34:15.420931    4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:34:15.445544 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:34:15.445517    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-7bhqj" event={"ID":"58693405-8a5b-4841-865d-aa29cc320e4d","Type":"ContainerStarted","Data":"1edf19ed8cd3c4cd4b9e246a49dbc154833867571d8ddf7025e5834697bc91bf"}
Oct 30 22:34:16.454571 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:34:16.454533    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-7bhqj" event={"ID":"58693405-8a5b-4841-865d-aa29cc320e4d","Type":"ContainerStarted","Data":"27913e02cf990f9b7065601c31973768d95c7cfc49c140fb17043f00795551a1"}
Oct 30 22:34:16.527670 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:34:16.527620    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-7bhqj" podStartSLOduration=2.527606871 podStartE2EDuration="2.527606871s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:34:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:34:16.516309559 +0000 UTC m=+1640.442235790" watchObservedRunningTime="2026-10-30 22:34:16.527606871 +0000 UTC m=+1640.453533110"
Oct 30 22:34:17.100070 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:34:17.098406    4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-7bhqj"
Oct 30 22:34:17.100390 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:34:17.100362    4477 patch_prober.go:29] interesting pod/e2e-inf-a-6922d3-openshift-default-689b5f95bc-7bhqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Oct 30 22:34:17.100481 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:34:17.100418    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-7bhqj" podUID="58693405-8a5b-4841-865d-aa29cc320e4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Oct 30 22:34:18.108100 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:34:18.107011    4477 patch_prober.go:29] interesting pod/e2e-inf-a-6922d3-openshift-default-689b5f95bc-7bhqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Oct 30 22:34:18.108100 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:34:18.107195    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-7bhqj" podUID="58693405-8a5b-4841-865d-aa29cc320e4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Oct 30 22:34:19.096061 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:34:19.095947    4477 patch_prober.go:29] interesting pod/e2e-inf-a-6922d3-openshift-default-689b5f95bc-7bhqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Oct 30 22:34:19.096061 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:34:19.096007    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-7bhqj" podUID="58693405-8a5b-4841-865d-aa29cc320e4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Oct 30 22:34:20.094747 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:34:20.094704    4477 patch_prober.go:29] interesting pod/e2e-inf-a-6922d3-openshift-default-689b5f95bc-7bhqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Oct 30 22:34:20.095298 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:34:20.094787    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-7bhqj" podUID="58693405-8a5b-4841-865d-aa29cc320e4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Oct 30 22:34:21.095492 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:34:21.095459    4477 patch_prober.go:29] interesting pod/e2e-inf-a-6922d3-openshift-default-689b5f95bc-7bhqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused" start-of-body=
Oct 30 22:34:21.095917 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:34:21.095511    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-7bhqj" podUID="58693405-8a5b-4841-865d-aa29cc320e4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.111:15021/healthz/ready\": dial tcp 10.134.0.111:15021: connect: connection refused"
Oct 30 22:34:22.100095 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:34:22.100069    4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-7bhqj"
Oct 30 22:34:22.100546 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:34:22.100532    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-7bhqj"
Oct 30 22:34:22.101308 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:34:22.101289    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-7bhqj"
Oct 30 22:34:24.129079 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:34:24.129027    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-de31dd-7d66978846-b464f"]
Oct 30 22:34:24.129524 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:34:24.129293    4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-de31dd-7d66978846-b464f" podUID="0f677752-d3f8-4af6-b50d-ca2b63559ca8" containerName="maas-api" containerID="cri-o://ea8ee8d06c436c491865df25e3c34c352401f1cb455458a3d18c13c53dd0bac3" gracePeriod=30
Oct 30 22:34:24.489102 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:34:24.483203    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-de31dd-7d66978846-b464f"
Oct 30 22:34:24.495303 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:34:24.493856    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f677752-d3f8-4af6-b50d-ca2b63559ca8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f677752-d3f8-4af6-b50d-ca2b63559ca8-metrics-certs\") pod \"0f677752-d3f8-4af6-b50d-ca2b63559ca8\" (UID: \"0f677752-d3f8-4af6-b50d-ca2b63559ca8\") "
Oct 30 22:34:24.495303 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:34:24.493930    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f677752-d3f8-4af6-b50d-ca2b63559ca8-kube-api-access-w2qvn\" (UniqueName: \"kubernetes.io/projected/0f677752-d3f8-4af6-b50d-ca2b63559ca8-kube-api-access-w2qvn\") pod \"0f677752-d3f8-4af6-b50d-ca2b63559ca8\" (UID: \"0f677752-d3f8-4af6-b50d-ca2b63559ca8\") "
Oct 30 22:34:24.495303 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:34:24.494025    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f677752-d3f8-4af6-b50d-ca2b63559ca8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f677752-d3f8-4af6-b50d-ca2b63559ca8-openshift-service-ca\") pod \"0f677752-d3f8-4af6-b50d-ca2b63559ca8\" (UID: \"0f677752-d3f8-4af6-b50d-ca2b63559ca8\") "
Oct 30 22:34:24.495303 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:34:24.494181    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f677752-d3f8-4af6-b50d-ca2b63559ca8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f677752-d3f8-4af6-b50d-ca2b63559ca8-maas-api-tls\") pod \"0f677752-d3f8-4af6-b50d-ca2b63559ca8\" (UID: \"0f677752-d3f8-4af6-b50d-ca2b63559ca8\") "
Oct 30 22:34:24.496823 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:34:24.496798    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f677752-d3f8-4af6-b50d-ca2b63559ca8-openshift-service-ca" pod "0f677752-d3f8-4af6-b50d-ca2b63559ca8" (UID: "0f677752-d3f8-4af6-b50d-ca2b63559ca8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:34:24.511549 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:34:24.511496    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f677752-d3f8-4af6-b50d-ca2b63559ca8-maas-api-tls" pod "0f677752-d3f8-4af6-b50d-ca2b63559ca8" (UID: "0f677752-d3f8-4af6-b50d-ca2b63559ca8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:34:24.518747 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:34:24.518703    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f677752-d3f8-4af6-b50d-ca2b63559ca8-kube-api-access-w2qvn" pod "0f677752-d3f8-4af6-b50d-ca2b63559ca8" (UID: "0f677752-d3f8-4af6-b50d-ca2b63559ca8"). InnerVolumeSpecName "kube-api-access-w2qvn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:34:24.529218 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:34:24.529172    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f677752-d3f8-4af6-b50d-ca2b63559ca8-metrics-certs" pod "0f677752-d3f8-4af6-b50d-ca2b63559ca8" (UID: "0f677752-d3f8-4af6-b50d-ca2b63559ca8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:34:24.574464 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:34:24.574430    4477 generic.go:356] "Generic (PLEG): container finished" podID="0f677752-d3f8-4af6-b50d-ca2b63559ca8" containerID="ea8ee8d06c436c491865df25e3c34c352401f1cb455458a3d18c13c53dd0bac3" exitCode=0
Oct 30 22:34:24.574618 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:34:24.574506    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-de31dd-7d66978846-b464f" event={"ID":"0f677752-d3f8-4af6-b50d-ca2b63559ca8","Type":"ContainerDied","Data":"ea8ee8d06c436c491865df25e3c34c352401f1cb455458a3d18c13c53dd0bac3"}
Oct 30 22:34:24.574618 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:34:24.574543    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-de31dd-7d66978846-b464f" event={"ID":"0f677752-d3f8-4af6-b50d-ca2b63559ca8","Type":"ContainerDied","Data":"e12534318403ec9ec763c33ccd216dacc73744dda87482c1b3969b8eb9cfec30"}
Oct 30 22:34:24.574618 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:34:24.574563    4477 scope.go:122] "RemoveContainer" containerID="ea8ee8d06c436c491865df25e3c34c352401f1cb455458a3d18c13c53dd0bac3"
Oct 30 22:34:24.574741 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:34:24.574695    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-de31dd-7d66978846-b464f"
Oct 30 22:34:24.597698 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:34:24.597665    4477 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f677752-d3f8-4af6-b50d-ca2b63559ca8-openshift-service-ca\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:34:24.597874 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:34:24.597860    4477 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f677752-d3f8-4af6-b50d-ca2b63559ca8-maas-api-tls\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:34:24.597965 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:34:24.597955    4477 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f677752-d3f8-4af6-b50d-ca2b63559ca8-metrics-certs\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:34:24.598171 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:34:24.598154    4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2qvn\" (UniqueName: \"kubernetes.io/projected/0f677752-d3f8-4af6-b50d-ca2b63559ca8-kube-api-access-w2qvn\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:34:24.615108 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:34:24.614865    4477 scope.go:122] "RemoveContainer" containerID="ea8ee8d06c436c491865df25e3c34c352401f1cb455458a3d18c13c53dd0bac3"
Oct 30 22:34:24.615402 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:34:24.615318    4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea8ee8d06c436c491865df25e3c34c352401f1cb455458a3d18c13c53dd0bac3\": container with ID starting with ea8ee8d06c436c491865df25e3c34c352401f1cb455458a3d18c13c53dd0bac3 not found: ID does not exist" containerID="ea8ee8d06c436c491865df25e3c34c352401f1cb455458a3d18c13c53dd0bac3"
Oct 30 22:34:24.615402 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:34:24.615358    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea8ee8d06c436c491865df25e3c34c352401f1cb455458a3d18c13c53dd0bac3"} err="failed to get container status \"ea8ee8d06c436c491865df25e3c34c352401f1cb455458a3d18c13c53dd0bac3\": rpc error: code = NotFound desc = could not find container \"ea8ee8d06c436c491865df25e3c34c352401f1cb455458a3d18c13c53dd0bac3\": container with ID starting with ea8ee8d06c436c491865df25e3c34c352401f1cb455458a3d18c13c53dd0bac3 not found: ID does not exist"
Oct 30 22:34:24.640305 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:34:24.640272    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-de31dd-7d66978846-b464f"]
Oct 30 22:34:24.646009 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:34:24.645981    4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-de31dd-7d66978846-b464f"]
Oct 30 22:34:26.194885 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:34:26.194852    4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f677752-d3f8-4af6-b50d-ca2b63559ca8" path="/var/lib/kubelet/pods/0f677752-d3f8-4af6-b50d-ca2b63559ca8/volumes"
Oct 30 22:34:26.276752 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:34:26.276711    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 22:34:53.077508 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:34:53.077411    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6bz92"]
Oct 30 22:34:56.267199 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:34:56.267166    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 22:34:59.891953 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:34:59.891921    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-txq8f"]
Oct 30 22:34:59.892606 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:34:59.892591    4477 cpu_manager.go:414] "containerMap: removing container" podUID="0f677752-d3f8-4af6-b50d-ca2b63559ca8" containerName="maas-api"
Oct 30 22:34:59.892658 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:34:59.892608    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f677752-d3f8-4af6-b50d-ca2b63559ca8" containerName="maas-api"
Oct 30 22:34:59.892759 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:34:59.892749    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f677752-d3f8-4af6-b50d-ca2b63559ca8" containerName="maas-api"
Oct 30 22:34:59.897732 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:34:59.897716    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-txq8f"
Oct 30 22:34:59.922158 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:34:59.922130    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-txq8f"]
Oct 30 22:34:59.965565 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:34:59.965529    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95030005-bb27-46a1-9238-503138ca97f6-istiod-ca-cert\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-txq8f\" (UID: \"95030005-bb27-46a1-9238-503138ca97f6\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-txq8f"
Oct 30 22:34:59.965720 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:34:59.965640    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95030005-bb27-46a1-9238-503138ca97f6-workload-certs\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-txq8f\" (UID: \"95030005-bb27-46a1-9238-503138ca97f6\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-txq8f"
Oct 30 22:34:59.965720 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:34:59.965695    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95030005-bb27-46a1-9238-503138ca97f6-istio-podinfo\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-txq8f\" (UID: \"95030005-bb27-46a1-9238-503138ca97f6\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-txq8f"
Oct 30 22:34:59.965836 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:34:59.965796    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmj89\" (UniqueName: \"kubernetes.io/projected/95030005-bb27-46a1-9238-503138ca97f6-kube-api-access-wmj89\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-txq8f\" (UID: \"95030005-bb27-46a1-9238-503138ca97f6\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-txq8f"
Oct 30 22:34:59.965898 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:34:59.965876    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95030005-bb27-46a1-9238-503138ca97f6-istio-token\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-txq8f\" (UID: \"95030005-bb27-46a1-9238-503138ca97f6\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-txq8f"
Oct 30 22:34:59.966032 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:34:59.966009    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95030005-bb27-46a1-9238-503138ca97f6-istio-envoy\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-txq8f\" (UID: \"95030005-bb27-46a1-9238-503138ca97f6\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-txq8f"
Oct 30 22:34:59.966118 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:34:59.966086    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95030005-bb27-46a1-9238-503138ca97f6-workload-socket\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-txq8f\" (UID: \"95030005-bb27-46a1-9238-503138ca97f6\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-txq8f"
Oct 30 22:34:59.966298 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:34:59.966270    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95030005-bb27-46a1-9238-503138ca97f6-istio-data\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-txq8f\" (UID: \"95030005-bb27-46a1-9238-503138ca97f6\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-txq8f"
Oct 30 22:34:59.966426 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:34:59.966330    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95030005-bb27-46a1-9238-503138ca97f6-credential-socket\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-txq8f\" (UID: \"95030005-bb27-46a1-9238-503138ca97f6\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-txq8f"
Oct 30 22:35:00.068257 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:00.068222    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95030005-bb27-46a1-9238-503138ca97f6-credential-socket\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-txq8f\" (UID: \"95030005-bb27-46a1-9238-503138ca97f6\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-txq8f"
Oct 30 22:35:00.068457 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:00.068286    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95030005-bb27-46a1-9238-503138ca97f6-istiod-ca-cert\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-txq8f\" (UID: \"95030005-bb27-46a1-9238-503138ca97f6\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-txq8f"
Oct 30 22:35:00.068457 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:00.068316    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95030005-bb27-46a1-9238-503138ca97f6-workload-certs\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-txq8f\" (UID: \"95030005-bb27-46a1-9238-503138ca97f6\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-txq8f"
Oct 30 22:35:00.068457 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:00.068348    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95030005-bb27-46a1-9238-503138ca97f6-istio-podinfo\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-txq8f\" (UID: \"95030005-bb27-46a1-9238-503138ca97f6\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-txq8f"
Oct 30 22:35:00.068457 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:00.068401    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmj89\" (UniqueName: \"kubernetes.io/projected/95030005-bb27-46a1-9238-503138ca97f6-kube-api-access-wmj89\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-txq8f\" (UID: \"95030005-bb27-46a1-9238-503138ca97f6\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-txq8f"
Oct 30 22:35:00.068691 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:00.068462    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95030005-bb27-46a1-9238-503138ca97f6-istio-token\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-txq8f\" (UID: \"95030005-bb27-46a1-9238-503138ca97f6\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-txq8f"
Oct 30 22:35:00.068691 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:00.068505    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95030005-bb27-46a1-9238-503138ca97f6-istio-envoy\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-txq8f\" (UID: \"95030005-bb27-46a1-9238-503138ca97f6\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-txq8f"
Oct 30 22:35:00.068691 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:00.068545    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95030005-bb27-46a1-9238-503138ca97f6-workload-socket\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-txq8f\" (UID: \"95030005-bb27-46a1-9238-503138ca97f6\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-txq8f"
Oct 30 22:35:00.068848 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:00.068720    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95030005-bb27-46a1-9238-503138ca97f6-istio-data\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-txq8f\" (UID: \"95030005-bb27-46a1-9238-503138ca97f6\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-txq8f"
Oct 30 22:35:00.068848 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:00.068748    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95030005-bb27-46a1-9238-503138ca97f6-credential-socket\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-txq8f\" (UID: \"95030005-bb27-46a1-9238-503138ca97f6\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-txq8f"
Oct 30 22:35:00.068848 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:00.068782    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95030005-bb27-46a1-9238-503138ca97f6-workload-certs\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-txq8f\" (UID: \"95030005-bb27-46a1-9238-503138ca97f6\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-txq8f"
Oct 30 22:35:00.069104 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:00.069081    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95030005-bb27-46a1-9238-503138ca97f6-workload-socket\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-txq8f\" (UID: \"95030005-bb27-46a1-9238-503138ca97f6\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-txq8f"
Oct 30 22:35:00.069193 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:00.069087    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95030005-bb27-46a1-9238-503138ca97f6-istio-data\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-txq8f\" (UID: \"95030005-bb27-46a1-9238-503138ca97f6\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-txq8f"
Oct 30 22:35:00.069258 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:00.069245    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95030005-bb27-46a1-9238-503138ca97f6-istiod-ca-cert\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-txq8f\" (UID: \"95030005-bb27-46a1-9238-503138ca97f6\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-txq8f"
Oct 30 22:35:00.070894 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:00.070871    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95030005-bb27-46a1-9238-503138ca97f6-istio-envoy\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-txq8f\" (UID: \"95030005-bb27-46a1-9238-503138ca97f6\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-txq8f"
Oct 30 22:35:00.071054 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:00.071023    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95030005-bb27-46a1-9238-503138ca97f6-istio-podinfo\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-txq8f\" (UID: \"95030005-bb27-46a1-9238-503138ca97f6\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-txq8f"
Oct 30 22:35:00.089807 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:00.089753    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95030005-bb27-46a1-9238-503138ca97f6-istio-token\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-txq8f\" (UID: \"95030005-bb27-46a1-9238-503138ca97f6\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-txq8f"
Oct 30 22:35:00.090192 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:00.090170    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmj89\" (UniqueName: \"kubernetes.io/projected/95030005-bb27-46a1-9238-503138ca97f6-kube-api-access-wmj89\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-txq8f\" (UID: \"95030005-bb27-46a1-9238-503138ca97f6\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-txq8f"
Oct 30 22:35:00.216526 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:00.216490    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-txq8f"
Oct 30 22:35:00.530952 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:00.530878    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-txq8f"]
Oct 30 22:35:00.533717 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:35:00.533681    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95030005_bb27_46a1_9238_503138ca97f6.slice/crio-2d0b83ff2af69adffc5f2f7b9fcd3d7184cec909bbf86bcf199a22ab74e87818 WatchSource:0}: Error finding container 2d0b83ff2af69adffc5f2f7b9fcd3d7184cec909bbf86bcf199a22ab74e87818: Status 404 returned error can't find the container with id 2d0b83ff2af69adffc5f2f7b9fcd3d7184cec909bbf86bcf199a22ab74e87818
Oct 30 22:35:00.536601 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:00.536565    4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:35:00.536712 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:00.536648    4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:35:00.536712 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:00.536695    4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:35:00.814421 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:00.814322    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-txq8f" event={"ID":"95030005-bb27-46a1-9238-503138ca97f6","Type":"ContainerStarted","Data":"22294cb010b5e5259f33a86d207c642f175cfb7afcf2d391740469eec302ee14"}
Oct 30 22:35:00.814421 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:00.814374    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-txq8f" event={"ID":"95030005-bb27-46a1-9238-503138ca97f6","Type":"ContainerStarted","Data":"2d0b83ff2af69adffc5f2f7b9fcd3d7184cec909bbf86bcf199a22ab74e87818"}
Oct 30 22:35:00.858403 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:00.854711    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-txq8f" podStartSLOduration=1.854690391 podStartE2EDuration="1.854690391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:34:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:35:00.851888022 +0000 UTC m=+1684.777814245" watchObservedRunningTime="2026-10-30 22:35:00.854690391 +0000 UTC m=+1684.780616597"
Oct 30 22:35:01.098283 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:01.098209    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-30edd7-5ff8d79b9d-2lqtr"]
Oct 30 22:35:01.098675 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:01.098400    4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-30edd7-5ff8d79b9d-2lqtr" podUID="d278732b-d633-4a53-b3ce-1a3fc9ad6790" containerName="maas-api" containerID="cri-o://56c1075fdef60130d43328bf6e9b32f1a45679617d5fe12d18b4a8d7acd38ff0" gracePeriod=30
Oct 30 22:35:01.217303 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:01.217271    4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-txq8f"
Oct 30 22:35:01.219416 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:01.219389    4477 patch_prober.go:29] interesting pod/e2e-inf-a-6922d3-openshift-default-689b5f95bc-txq8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body=
Oct 30 22:35:01.219536 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:01.219448    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-txq8f" podUID="95030005-bb27-46a1-9238-503138ca97f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused"
Oct 30 22:35:01.565339 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:01.565298    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-30edd7-5ff8d79b9d-2lqtr"
Oct 30 22:35:01.632893 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:01.632855    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d278732b-d633-4a53-b3ce-1a3fc9ad6790-kube-api-access-k2577\" (UniqueName: \"kubernetes.io/projected/d278732b-d633-4a53-b3ce-1a3fc9ad6790-kube-api-access-k2577\") pod \"d278732b-d633-4a53-b3ce-1a3fc9ad6790\" (UID: \"d278732b-d633-4a53-b3ce-1a3fc9ad6790\") "
Oct 30 22:35:01.633819 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:01.633797    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d278732b-d633-4a53-b3ce-1a3fc9ad6790-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d278732b-d633-4a53-b3ce-1a3fc9ad6790-metrics-certs\") pod \"d278732b-d633-4a53-b3ce-1a3fc9ad6790\" (UID: \"d278732b-d633-4a53-b3ce-1a3fc9ad6790\") "
Oct 30 22:35:01.634472 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:01.634451    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d278732b-d633-4a53-b3ce-1a3fc9ad6790-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d278732b-d633-4a53-b3ce-1a3fc9ad6790-openshift-service-ca\") pod \"d278732b-d633-4a53-b3ce-1a3fc9ad6790\" (UID: \"d278732b-d633-4a53-b3ce-1a3fc9ad6790\") "
Oct 30 22:35:01.680558 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:01.637276    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d278732b-d633-4a53-b3ce-1a3fc9ad6790-kube-api-access-k2577" pod "d278732b-d633-4a53-b3ce-1a3fc9ad6790" (UID: "d278732b-d633-4a53-b3ce-1a3fc9ad6790"). InnerVolumeSpecName "kube-api-access-k2577". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:35:01.681033 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:01.665288    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d278732b-d633-4a53-b3ce-1a3fc9ad6790-openshift-service-ca" pod "d278732b-d633-4a53-b3ce-1a3fc9ad6790" (UID: "d278732b-d633-4a53-b3ce-1a3fc9ad6790"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:35:01.681281 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:01.679226    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d278732b-d633-4a53-b3ce-1a3fc9ad6790-metrics-certs" pod "d278732b-d633-4a53-b3ce-1a3fc9ad6790" (UID: "d278732b-d633-4a53-b3ce-1a3fc9ad6790"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:35:01.681802 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:01.681777    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d278732b-d633-4a53-b3ce-1a3fc9ad6790-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d278732b-d633-4a53-b3ce-1a3fc9ad6790-maas-api-tls\") pod \"d278732b-d633-4a53-b3ce-1a3fc9ad6790\" (UID: \"d278732b-d633-4a53-b3ce-1a3fc9ad6790\") "
Oct 30 22:35:01.700065 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:01.695854    4477 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d278732b-d633-4a53-b3ce-1a3fc9ad6790-openshift-service-ca\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:35:01.700065 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:01.695919    4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2577\" (UniqueName: \"kubernetes.io/projected/d278732b-d633-4a53-b3ce-1a3fc9ad6790-kube-api-access-k2577\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:35:01.700065 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:01.695935    4477 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d278732b-d633-4a53-b3ce-1a3fc9ad6790-metrics-certs\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:35:01.764118 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:01.749525    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d278732b-d633-4a53-b3ce-1a3fc9ad6790-maas-api-tls" pod "d278732b-d633-4a53-b3ce-1a3fc9ad6790" (UID: "d278732b-d633-4a53-b3ce-1a3fc9ad6790"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:35:01.819753 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:01.817554    4477 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d278732b-d633-4a53-b3ce-1a3fc9ad6790-maas-api-tls\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:35:01.827024 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:01.826994    4477 generic.go:356] "Generic (PLEG): container finished" podID="d278732b-d633-4a53-b3ce-1a3fc9ad6790" containerID="56c1075fdef60130d43328bf6e9b32f1a45679617d5fe12d18b4a8d7acd38ff0" exitCode=0
Oct 30 22:35:01.827209 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:01.827035    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-30edd7-5ff8d79b9d-2lqtr"
Oct 30 22:35:01.827209 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:01.827129    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-30edd7-5ff8d79b9d-2lqtr" event={"ID":"d278732b-d633-4a53-b3ce-1a3fc9ad6790","Type":"ContainerDied","Data":"56c1075fdef60130d43328bf6e9b32f1a45679617d5fe12d18b4a8d7acd38ff0"}
Oct 30 22:35:01.827209 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:01.827205    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-30edd7-5ff8d79b9d-2lqtr" event={"ID":"d278732b-d633-4a53-b3ce-1a3fc9ad6790","Type":"ContainerDied","Data":"721756434bca1d0b39fefc84c214134f6f734419407cf2c76eef8e9696708f08"}
Oct 30 22:35:01.827370 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:01.827235    4477 scope.go:122] "RemoveContainer" containerID="56c1075fdef60130d43328bf6e9b32f1a45679617d5fe12d18b4a8d7acd38ff0"
Oct 30 22:35:01.841991 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:01.841968    4477 scope.go:122] "RemoveContainer" containerID="56c1075fdef60130d43328bf6e9b32f1a45679617d5fe12d18b4a8d7acd38ff0"
Oct 30 22:35:01.842286 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:35:01.842265    4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56c1075fdef60130d43328bf6e9b32f1a45679617d5fe12d18b4a8d7acd38ff0\": container with ID starting with 56c1075fdef60130d43328bf6e9b32f1a45679617d5fe12d18b4a8d7acd38ff0 not found: ID does not exist" containerID="56c1075fdef60130d43328bf6e9b32f1a45679617d5fe12d18b4a8d7acd38ff0"
Oct 30 22:35:01.842347 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:01.842297    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56c1075fdef60130d43328bf6e9b32f1a45679617d5fe12d18b4a8d7acd38ff0"} err="failed to get container status \"56c1075fdef60130d43328bf6e9b32f1a45679617d5fe12d18b4a8d7acd38ff0\": rpc error: code = NotFound desc = could not find container \"56c1075fdef60130d43328bf6e9b32f1a45679617d5fe12d18b4a8d7acd38ff0\": container with ID starting with 56c1075fdef60130d43328bf6e9b32f1a45679617d5fe12d18b4a8d7acd38ff0 not found: ID does not exist"
Oct 30 22:35:01.898194 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:01.898142    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-30edd7-5ff8d79b9d-2lqtr"]
Oct 30 22:35:01.901592 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:01.901566    4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-30edd7-5ff8d79b9d-2lqtr"]
Oct 30 22:35:02.204991 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:02.204960    4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d278732b-d633-4a53-b3ce-1a3fc9ad6790" path="/var/lib/kubelet/pods/d278732b-d633-4a53-b3ce-1a3fc9ad6790/volumes"
Oct 30 22:35:02.216922 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:02.216892    4477 patch_prober.go:29] interesting pod/e2e-inf-a-6922d3-openshift-default-689b5f95bc-txq8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body=
Oct 30 22:35:02.217133 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:02.216945    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-txq8f" podUID="95030005-bb27-46a1-9238-503138ca97f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused"
Oct 30 22:35:03.217260 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:03.217229    4477 patch_prober.go:29] interesting pod/e2e-inf-a-6922d3-openshift-default-689b5f95bc-txq8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body=
Oct 30 22:35:03.217723 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:03.217288    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-txq8f" podUID="95030005-bb27-46a1-9238-503138ca97f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused"
Oct 30 22:35:04.217747 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:04.217711    4477 patch_prober.go:29] interesting pod/e2e-inf-a-6922d3-openshift-default-689b5f95bc-txq8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body=
Oct 30 22:35:04.218152 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:04.217773    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-txq8f" podUID="95030005-bb27-46a1-9238-503138ca97f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused"
Oct 30 22:35:05.217738 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:05.217706    4477 patch_prober.go:29] interesting pod/e2e-inf-a-6922d3-openshift-default-689b5f95bc-txq8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body=
Oct 30 22:35:05.217901 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:05.217756    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-txq8f" podUID="95030005-bb27-46a1-9238-503138ca97f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused"
Oct 30 22:35:06.217765 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:06.217730    4477 patch_prober.go:29] interesting pod/e2e-inf-a-6922d3-openshift-default-689b5f95bc-txq8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused" start-of-body=
Oct 30 22:35:06.218069 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:06.218029    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-txq8f" podUID="95030005-bb27-46a1-9238-503138ca97f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.112:15021/healthz/ready\": dial tcp 10.134.0.112:15021: connect: connection refused"
Oct 30 22:35:07.239514 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:07.239483    4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-txq8f"
Oct 30 22:35:07.240002 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:07.239811    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-txq8f"
Oct 30 22:35:07.240629 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:07.240601    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-txq8f"
Oct 30 22:35:09.480696 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:09.480660    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bf9c2f-cfd8bcd86-pdn6v"]
Oct 30 22:35:09.481284 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:09.480887    4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bf9c2f-cfd8bcd86-pdn6v" podUID="b450d906-242d-4c3e-97cd-1e805041f85a" containerName="maas-api" containerID="cri-o://f789827d6171ac05f2a5f959b789997ebe0470542314912708fa9470abca99bc" gracePeriod=30
Oct 30 22:35:09.843445 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:09.843413    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bf9c2f-cfd8bcd86-pdn6v"
Oct 30 22:35:09.899637 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:09.898944    4477 generic.go:356] "Generic (PLEG): container finished" podID="b450d906-242d-4c3e-97cd-1e805041f85a" containerID="f789827d6171ac05f2a5f959b789997ebe0470542314912708fa9470abca99bc" exitCode=0
Oct 30 22:35:09.899637 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:09.899006    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bf9c2f-cfd8bcd86-pdn6v" event={"ID":"b450d906-242d-4c3e-97cd-1e805041f85a","Type":"ContainerDied","Data":"f789827d6171ac05f2a5f959b789997ebe0470542314912708fa9470abca99bc"}
Oct 30 22:35:09.899637 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:09.899071    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bf9c2f-cfd8bcd86-pdn6v" event={"ID":"b450d906-242d-4c3e-97cd-1e805041f85a","Type":"ContainerDied","Data":"a541c7e442d3965129509ab110ad25b733af8ea1098f702def9511893b2e9b5d"}
Oct 30 22:35:09.899637 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:09.899102    4477 scope.go:122] "RemoveContainer" containerID="f789827d6171ac05f2a5f959b789997ebe0470542314912708fa9470abca99bc"
Oct 30 22:35:09.899637 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:09.899257    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-bf9c2f-cfd8bcd86-pdn6v"
Oct 30 22:35:09.920218 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:09.920193    4477 scope.go:122] "RemoveContainer" containerID="f789827d6171ac05f2a5f959b789997ebe0470542314912708fa9470abca99bc"
Oct 30 22:35:09.920765 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:35:09.920741    4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f789827d6171ac05f2a5f959b789997ebe0470542314912708fa9470abca99bc\": container with ID starting with f789827d6171ac05f2a5f959b789997ebe0470542314912708fa9470abca99bc not found: ID does not exist" containerID="f789827d6171ac05f2a5f959b789997ebe0470542314912708fa9470abca99bc"
Oct 30 22:35:09.920930 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:09.920904    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f789827d6171ac05f2a5f959b789997ebe0470542314912708fa9470abca99bc"} err="failed to get container status \"f789827d6171ac05f2a5f959b789997ebe0470542314912708fa9470abca99bc\": rpc error: code = NotFound desc = could not find container \"f789827d6171ac05f2a5f959b789997ebe0470542314912708fa9470abca99bc\": container with ID starting with f789827d6171ac05f2a5f959b789997ebe0470542314912708fa9470abca99bc not found: ID does not exist"
Oct 30 22:35:09.957606 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:09.957569    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b450d906-242d-4c3e-97cd-1e805041f85a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b450d906-242d-4c3e-97cd-1e805041f85a-maas-api-tls\") pod \"b450d906-242d-4c3e-97cd-1e805041f85a\" (UID: \"b450d906-242d-4c3e-97cd-1e805041f85a\") "
Oct 30 22:35:09.957925 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:09.957865    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b450d906-242d-4c3e-97cd-1e805041f85a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b450d906-242d-4c3e-97cd-1e805041f85a-metrics-certs\") pod \"b450d906-242d-4c3e-97cd-1e805041f85a\" (UID: \"b450d906-242d-4c3e-97cd-1e805041f85a\") "
Oct 30 22:35:09.958438 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:09.958402    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b450d906-242d-4c3e-97cd-1e805041f85a-kube-api-access-hrsgb\" (UniqueName: \"kubernetes.io/projected/b450d906-242d-4c3e-97cd-1e805041f85a-kube-api-access-hrsgb\") pod \"b450d906-242d-4c3e-97cd-1e805041f85a\" (UID: \"b450d906-242d-4c3e-97cd-1e805041f85a\") "
Oct 30 22:35:09.958558 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:09.958493    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b450d906-242d-4c3e-97cd-1e805041f85a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b450d906-242d-4c3e-97cd-1e805041f85a-openshift-service-ca\") pod \"b450d906-242d-4c3e-97cd-1e805041f85a\" (UID: \"b450d906-242d-4c3e-97cd-1e805041f85a\") "
Oct 30 22:35:09.960598 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:09.960568    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b450d906-242d-4c3e-97cd-1e805041f85a-openshift-service-ca" pod "b450d906-242d-4c3e-97cd-1e805041f85a" (UID: "b450d906-242d-4c3e-97cd-1e805041f85a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:35:09.962192 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:09.962164    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b450d906-242d-4c3e-97cd-1e805041f85a-metrics-certs" pod "b450d906-242d-4c3e-97cd-1e805041f85a" (UID: "b450d906-242d-4c3e-97cd-1e805041f85a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:35:09.967024 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:09.966994    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b450d906-242d-4c3e-97cd-1e805041f85a-maas-api-tls" pod "b450d906-242d-4c3e-97cd-1e805041f85a" (UID: "b450d906-242d-4c3e-97cd-1e805041f85a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:35:09.967301 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:09.967277    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b450d906-242d-4c3e-97cd-1e805041f85a-kube-api-access-hrsgb" pod "b450d906-242d-4c3e-97cd-1e805041f85a" (UID: "b450d906-242d-4c3e-97cd-1e805041f85a"). InnerVolumeSpecName "kube-api-access-hrsgb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:35:10.062434 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:10.062349    4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrsgb\" (UniqueName: \"kubernetes.io/projected/b450d906-242d-4c3e-97cd-1e805041f85a-kube-api-access-hrsgb\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:35:10.062434 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:10.062389    4477 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b450d906-242d-4c3e-97cd-1e805041f85a-openshift-service-ca\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:35:10.062434 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:10.062402    4477 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b450d906-242d-4c3e-97cd-1e805041f85a-maas-api-tls\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:35:10.062434 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:10.062414    4477 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b450d906-242d-4c3e-97cd-1e805041f85a-metrics-certs\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:35:10.254578 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:10.254542    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bf9c2f-cfd8bcd86-pdn6v"]
Oct 30 22:35:10.258813 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:10.258258    4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-bf9c2f-cfd8bcd86-pdn6v"]
Oct 30 22:35:12.195014 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:12.194982    4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b450d906-242d-4c3e-97cd-1e805041f85a" path="/var/lib/kubelet/pods/b450d906-242d-4c3e-97cd-1e805041f85a/volumes"
Oct 30 22:35:26.285098 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:35:26.285020    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 22:35:29.887393 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:29.887356    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-7a1ee7-787c657595-2vdgl"]
Oct 30 22:35:29.887884 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:29.887597    4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7a1ee7-787c657595-2vdgl" podUID="ffd9f8a7-2785-4a15-8121-ff13661dadb1" containerName="maas-api" containerID="cri-o://499fc8694791b5281e6e8a00fe488a9aff57f028e795aafa03e10f8c1322d598" gracePeriod=30
Oct 30 22:35:30.100590 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:30.100553    4477 generic.go:356] "Generic (PLEG): container finished" podID="ffd9f8a7-2785-4a15-8121-ff13661dadb1" containerID="499fc8694791b5281e6e8a00fe488a9aff57f028e795aafa03e10f8c1322d598" exitCode=0
Oct 30 22:35:30.100794 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:30.100611    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7a1ee7-787c657595-2vdgl" event={"ID":"ffd9f8a7-2785-4a15-8121-ff13661dadb1","Type":"ContainerDied","Data":"499fc8694791b5281e6e8a00fe488a9aff57f028e795aafa03e10f8c1322d598"}
Oct 30 22:35:30.207979 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:30.207950    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7a1ee7-787c657595-2vdgl"
Oct 30 22:35:30.385063 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:30.384365    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffd9f8a7-2785-4a15-8121-ff13661dadb1-kube-api-access-w76k5\" (UniqueName: \"kubernetes.io/projected/ffd9f8a7-2785-4a15-8121-ff13661dadb1-kube-api-access-w76k5\") pod \"ffd9f8a7-2785-4a15-8121-ff13661dadb1\" (UID: \"ffd9f8a7-2785-4a15-8121-ff13661dadb1\") "
Oct 30 22:35:30.385063 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:30.384475    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffd9f8a7-2785-4a15-8121-ff13661dadb1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffd9f8a7-2785-4a15-8121-ff13661dadb1-openshift-service-ca\") pod \"ffd9f8a7-2785-4a15-8121-ff13661dadb1\" (UID: \"ffd9f8a7-2785-4a15-8121-ff13661dadb1\") "
Oct 30 22:35:30.385063 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:30.384592    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ffd9f8a7-2785-4a15-8121-ff13661dadb1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffd9f8a7-2785-4a15-8121-ff13661dadb1-metrics-certs\") pod \"ffd9f8a7-2785-4a15-8121-ff13661dadb1\" (UID: \"ffd9f8a7-2785-4a15-8121-ff13661dadb1\") "
Oct 30 22:35:30.385063 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:30.384626    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ffd9f8a7-2785-4a15-8121-ff13661dadb1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ffd9f8a7-2785-4a15-8121-ff13661dadb1-maas-api-tls\") pod \"ffd9f8a7-2785-4a15-8121-ff13661dadb1\" (UID: \"ffd9f8a7-2785-4a15-8121-ff13661dadb1\") "
Oct 30 22:35:30.404068 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:30.395550    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffd9f8a7-2785-4a15-8121-ff13661dadb1-kube-api-access-w76k5" pod "ffd9f8a7-2785-4a15-8121-ff13661dadb1" (UID: "ffd9f8a7-2785-4a15-8121-ff13661dadb1"). InnerVolumeSpecName "kube-api-access-w76k5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:35:30.404068 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:30.398114    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffd9f8a7-2785-4a15-8121-ff13661dadb1-metrics-certs" pod "ffd9f8a7-2785-4a15-8121-ff13661dadb1" (UID: "ffd9f8a7-2785-4a15-8121-ff13661dadb1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:35:30.404395 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:30.404244    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffd9f8a7-2785-4a15-8121-ff13661dadb1-openshift-service-ca" pod "ffd9f8a7-2785-4a15-8121-ff13661dadb1" (UID: "ffd9f8a7-2785-4a15-8121-ff13661dadb1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:35:30.409441 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:30.404441    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffd9f8a7-2785-4a15-8121-ff13661dadb1-maas-api-tls" pod "ffd9f8a7-2785-4a15-8121-ff13661dadb1" (UID: "ffd9f8a7-2785-4a15-8121-ff13661dadb1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:35:30.487117 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:30.487087    4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w76k5\" (UniqueName: \"kubernetes.io/projected/ffd9f8a7-2785-4a15-8121-ff13661dadb1-kube-api-access-w76k5\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:35:30.487117 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:30.487115    4477 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ffd9f8a7-2785-4a15-8121-ff13661dadb1-openshift-service-ca\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:35:30.487361 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:30.487126    4477 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ffd9f8a7-2785-4a15-8121-ff13661dadb1-metrics-certs\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:35:30.487361 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:30.487137    4477 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ffd9f8a7-2785-4a15-8121-ff13661dadb1-maas-api-tls\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:35:31.107948 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:31.107920    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7a1ee7-787c657595-2vdgl"
Oct 30 22:35:31.108445 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:31.107949    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-7a1ee7-787c657595-2vdgl" event={"ID":"ffd9f8a7-2785-4a15-8121-ff13661dadb1","Type":"ContainerDied","Data":"4bb8688ce73ad33739c1c9ac4c33b01fafb070509121a60a920be92ae9bf95ff"}
Oct 30 22:35:31.108445 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:31.108007    4477 scope.go:122] "RemoveContainer" containerID="499fc8694791b5281e6e8a00fe488a9aff57f028e795aafa03e10f8c1322d598"
Oct 30 22:35:31.154601 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:31.154572    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-7a1ee7-787c657595-2vdgl"]
Oct 30 22:35:31.164507 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:31.164483    4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-7a1ee7-787c657595-2vdgl"]
Oct 30 22:35:32.194726 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:32.194692    4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffd9f8a7-2785-4a15-8121-ff13661dadb1" path="/var/lib/kubelet/pods/ffd9f8a7-2785-4a15-8121-ff13661dadb1/volumes"
Oct 30 22:35:36.034006 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:36.033973    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-3d38f6-74d6675f9d-sflzt"]
Oct 30 22:35:36.034493 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:36.034246    4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3d38f6-74d6675f9d-sflzt" podUID="46d350c3-9eff-403a-844d-f50edc815b45" containerName="maas-api" containerID="cri-o://db425b478f13991d695f8d1ac5424aef32358ea4c399c95d080e8bc0b9675960" gracePeriod=30
Oct 30 22:35:36.370474 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:36.370336    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3d38f6-74d6675f9d-sflzt"
Oct 30 22:35:36.463740 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:36.461239    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46d350c3-9eff-403a-844d-f50edc815b45-metrics-certs\" (UniqueName: \"kubernetes.io/secret/46d350c3-9eff-403a-844d-f50edc815b45-metrics-certs\") pod \"46d350c3-9eff-403a-844d-f50edc815b45\" (UID: \"46d350c3-9eff-403a-844d-f50edc815b45\") "
Oct 30 22:35:36.463740 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:36.461327    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46d350c3-9eff-403a-844d-f50edc815b45-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46d350c3-9eff-403a-844d-f50edc815b45-openshift-service-ca\") pod \"46d350c3-9eff-403a-844d-f50edc815b45\" (UID: \"46d350c3-9eff-403a-844d-f50edc815b45\") "
Oct 30 22:35:36.463740 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:36.461374    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46d350c3-9eff-403a-844d-f50edc815b45-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46d350c3-9eff-403a-844d-f50edc815b45-maas-api-tls\") pod \"46d350c3-9eff-403a-844d-f50edc815b45\" (UID: \"46d350c3-9eff-403a-844d-f50edc815b45\") "
Oct 30 22:35:36.463740 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:36.461415    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46d350c3-9eff-403a-844d-f50edc815b45-kube-api-access-q6gdm\" (UniqueName: \"kubernetes.io/projected/46d350c3-9eff-403a-844d-f50edc815b45-kube-api-access-q6gdm\") pod \"46d350c3-9eff-403a-844d-f50edc815b45\" (UID: \"46d350c3-9eff-403a-844d-f50edc815b45\") "
Oct 30 22:35:36.463740 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:36.461945    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46d350c3-9eff-403a-844d-f50edc815b45-openshift-service-ca" pod "46d350c3-9eff-403a-844d-f50edc815b45" (UID: "46d350c3-9eff-403a-844d-f50edc815b45"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:35:36.463740 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:36.463639    4477 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46d350c3-9eff-403a-844d-f50edc815b45-openshift-service-ca\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:35:36.472262 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:36.472223    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46d350c3-9eff-403a-844d-f50edc815b45-metrics-certs" pod "46d350c3-9eff-403a-844d-f50edc815b45" (UID: "46d350c3-9eff-403a-844d-f50edc815b45"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:35:36.472399 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:36.472263    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46d350c3-9eff-403a-844d-f50edc815b45-kube-api-access-q6gdm" pod "46d350c3-9eff-403a-844d-f50edc815b45" (UID: "46d350c3-9eff-403a-844d-f50edc815b45"). InnerVolumeSpecName "kube-api-access-q6gdm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:35:36.472578 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:36.472556    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46d350c3-9eff-403a-844d-f50edc815b45-maas-api-tls" pod "46d350c3-9eff-403a-844d-f50edc815b45" (UID: "46d350c3-9eff-403a-844d-f50edc815b45"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:35:36.568063 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:36.565036    4477 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46d350c3-9eff-403a-844d-f50edc815b45-metrics-certs\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:35:36.568063 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:36.565090    4477 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46d350c3-9eff-403a-844d-f50edc815b45-maas-api-tls\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:35:36.568063 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:36.565105    4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6gdm\" (UniqueName: \"kubernetes.io/projected/46d350c3-9eff-403a-844d-f50edc815b45-kube-api-access-q6gdm\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:35:37.152012 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:37.151978    4477 generic.go:356] "Generic (PLEG): container finished" podID="46d350c3-9eff-403a-844d-f50edc815b45" containerID="db425b478f13991d695f8d1ac5424aef32358ea4c399c95d080e8bc0b9675960" exitCode=0
Oct 30 22:35:37.152439 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:37.152031    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3d38f6-74d6675f9d-sflzt"
Oct 30 22:35:37.152439 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:37.152074    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3d38f6-74d6675f9d-sflzt" event={"ID":"46d350c3-9eff-403a-844d-f50edc815b45","Type":"ContainerDied","Data":"db425b478f13991d695f8d1ac5424aef32358ea4c399c95d080e8bc0b9675960"}
Oct 30 22:35:37.152439 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:37.152130    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-3d38f6-74d6675f9d-sflzt" event={"ID":"46d350c3-9eff-403a-844d-f50edc815b45","Type":"ContainerDied","Data":"c57ec7ba044e6ff7ccbd291a48aa3ed81ec24a0ffd6b855838e033d9e92cf1ac"}
Oct 30 22:35:37.152439 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:37.152152    4477 scope.go:122] "RemoveContainer" containerID="db425b478f13991d695f8d1ac5424aef32358ea4c399c95d080e8bc0b9675960"
Oct 30 22:35:37.162955 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:37.162936    4477 scope.go:122] "RemoveContainer" containerID="db425b478f13991d695f8d1ac5424aef32358ea4c399c95d080e8bc0b9675960"
Oct 30 22:35:37.163232 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:35:37.163211    4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db425b478f13991d695f8d1ac5424aef32358ea4c399c95d080e8bc0b9675960\": container with ID starting with db425b478f13991d695f8d1ac5424aef32358ea4c399c95d080e8bc0b9675960 not found: ID does not exist" containerID="db425b478f13991d695f8d1ac5424aef32358ea4c399c95d080e8bc0b9675960"
Oct 30 22:35:37.163293 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:37.163239    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db425b478f13991d695f8d1ac5424aef32358ea4c399c95d080e8bc0b9675960"} err="failed to get container status \"db425b478f13991d695f8d1ac5424aef32358ea4c399c95d080e8bc0b9675960\": rpc error: code = NotFound desc = could not find container \"db425b478f13991d695f8d1ac5424aef32358ea4c399c95d080e8bc0b9675960\": container with ID starting with db425b478f13991d695f8d1ac5424aef32358ea4c399c95d080e8bc0b9675960 not found: ID does not exist"
Oct 30 22:35:37.199349 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:37.199325    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-3d38f6-74d6675f9d-sflzt"]
Oct 30 22:35:37.209213 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:37.209190    4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-3d38f6-74d6675f9d-sflzt"]
Oct 30 22:35:38.194785 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:38.194753    4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46d350c3-9eff-403a-844d-f50edc815b45" path="/var/lib/kubelet/pods/46d350c3-9eff-403a-844d-f50edc815b45/volumes"
Oct 30 22:35:42.105724 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:42.105690    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6bz92"]
Oct 30 22:35:54.845304 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:54.845273    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-6922d3/test-model-6922d3-kserve-65bc85fbc-tcdmn"]
Oct 30 22:35:54.845901 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:54.845885    4477 cpu_manager.go:414] "containerMap: removing container" podUID="d278732b-d633-4a53-b3ce-1a3fc9ad6790" containerName="maas-api"
Oct 30 22:35:54.845901 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:54.845901    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d278732b-d633-4a53-b3ce-1a3fc9ad6790" containerName="maas-api"
Oct 30 22:35:54.846022 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:54.845913    4477 cpu_manager.go:414] "containerMap: removing container" podUID="46d350c3-9eff-403a-844d-f50edc815b45" containerName="maas-api"
Oct 30 22:35:54.846022 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:54.845919    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46d350c3-9eff-403a-844d-f50edc815b45" containerName="maas-api"
Oct 30 22:35:54.846022 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:54.845948    4477 cpu_manager.go:414] "containerMap: removing container" podUID="b450d906-242d-4c3e-97cd-1e805041f85a" containerName="maas-api"
Oct 30 22:35:54.846022 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:54.845953    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b450d906-242d-4c3e-97cd-1e805041f85a" containerName="maas-api"
Oct 30 22:35:54.846022 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:54.845966    4477 cpu_manager.go:414] "containerMap: removing container" podUID="ffd9f8a7-2785-4a15-8121-ff13661dadb1" containerName="maas-api"
Oct 30 22:35:54.846022 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:54.845970    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffd9f8a7-2785-4a15-8121-ff13661dadb1" containerName="maas-api"
Oct 30 22:35:54.846216 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:54.846089    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="d278732b-d633-4a53-b3ce-1a3fc9ad6790" containerName="maas-api"
Oct 30 22:35:54.846216 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:54.846097    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="46d350c3-9eff-403a-844d-f50edc815b45" containerName="maas-api"
Oct 30 22:35:54.846216 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:54.846111    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="b450d906-242d-4c3e-97cd-1e805041f85a" containerName="maas-api"
Oct 30 22:35:54.846216 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:54.846119    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffd9f8a7-2785-4a15-8121-ff13661dadb1" containerName="maas-api"
Oct 30 22:35:54.849815 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:54.849793    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-6922d3/test-model-6922d3-kserve-65bc85fbc-tcdmn"
Oct 30 22:35:54.873337 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:54.873311    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-6922d3\"/\"test-model-6922d3-kserve-self-signed-certs\""
Oct 30 22:35:54.873586 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:54.873568    4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-6922d3\"/\"kube-root-ca.crt\""
Oct 30 22:35:54.873865 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:54.873841    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-6922d3\"/\"default-dockercfg-8rmt5\""
Oct 30 22:35:54.879175 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:54.879149    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-6922d3/test-model-6922d3-kserve-65bc85fbc-tcdmn"]
Oct 30 22:35:54.890459 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:54.890434    4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-6922d3\"/\"openshift-service-ca.crt\""
Oct 30 22:35:54.948600 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:54.948567    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cccc800e-68e3-40ff-b7a0-29f5a777e83b-home\") pod \"test-model-6922d3-kserve-65bc85fbc-tcdmn\" (UID: \"cccc800e-68e3-40ff-b7a0-29f5a777e83b\") " pod="ai-tenant-e2e-inf-a-6922d3/test-model-6922d3-kserve-65bc85fbc-tcdmn"
Oct 30 22:35:54.948824 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:54.948798    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cccc800e-68e3-40ff-b7a0-29f5a777e83b-model-cache\") pod \"test-model-6922d3-kserve-65bc85fbc-tcdmn\" (UID: \"cccc800e-68e3-40ff-b7a0-29f5a777e83b\") " pod="ai-tenant-e2e-inf-a-6922d3/test-model-6922d3-kserve-65bc85fbc-tcdmn"
Oct 30 22:35:54.948939 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:54.948879    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cccc800e-68e3-40ff-b7a0-29f5a777e83b-tls-certs\") pod \"test-model-6922d3-kserve-65bc85fbc-tcdmn\" (UID: \"cccc800e-68e3-40ff-b7a0-29f5a777e83b\") " pod="ai-tenant-e2e-inf-a-6922d3/test-model-6922d3-kserve-65bc85fbc-tcdmn"
Oct 30 22:35:54.948939 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:54.948929    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cccc800e-68e3-40ff-b7a0-29f5a777e83b-dshm\") pod \"test-model-6922d3-kserve-65bc85fbc-tcdmn\" (UID: \"cccc800e-68e3-40ff-b7a0-29f5a777e83b\") " pod="ai-tenant-e2e-inf-a-6922d3/test-model-6922d3-kserve-65bc85fbc-tcdmn"
Oct 30 22:35:54.949107 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:54.949088    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5vrx\" (UniqueName: \"kubernetes.io/projected/cccc800e-68e3-40ff-b7a0-29f5a777e83b-kube-api-access-l5vrx\") pod \"test-model-6922d3-kserve-65bc85fbc-tcdmn\" (UID: \"cccc800e-68e3-40ff-b7a0-29f5a777e83b\") " pod="ai-tenant-e2e-inf-a-6922d3/test-model-6922d3-kserve-65bc85fbc-tcdmn"
Oct 30 22:35:54.949155 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:54.949130    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cccc800e-68e3-40ff-b7a0-29f5a777e83b-tmp-dir\") pod \"test-model-6922d3-kserve-65bc85fbc-tcdmn\" (UID: \"cccc800e-68e3-40ff-b7a0-29f5a777e83b\") " pod="ai-tenant-e2e-inf-a-6922d3/test-model-6922d3-kserve-65bc85fbc-tcdmn"
Oct 30 22:35:55.050983 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:55.050946    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cccc800e-68e3-40ff-b7a0-29f5a777e83b-tls-certs\") pod \"test-model-6922d3-kserve-65bc85fbc-tcdmn\" (UID: \"cccc800e-68e3-40ff-b7a0-29f5a777e83b\") " pod="ai-tenant-e2e-inf-a-6922d3/test-model-6922d3-kserve-65bc85fbc-tcdmn"
Oct 30 22:35:55.051193 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:55.051025    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cccc800e-68e3-40ff-b7a0-29f5a777e83b-dshm\") pod \"test-model-6922d3-kserve-65bc85fbc-tcdmn\" (UID: \"cccc800e-68e3-40ff-b7a0-29f5a777e83b\") " pod="ai-tenant-e2e-inf-a-6922d3/test-model-6922d3-kserve-65bc85fbc-tcdmn"
Oct 30 22:35:55.051193 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:55.051181    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5vrx\" (UniqueName: \"kubernetes.io/projected/cccc800e-68e3-40ff-b7a0-29f5a777e83b-kube-api-access-l5vrx\") pod \"test-model-6922d3-kserve-65bc85fbc-tcdmn\" (UID: \"cccc800e-68e3-40ff-b7a0-29f5a777e83b\") " pod="ai-tenant-e2e-inf-a-6922d3/test-model-6922d3-kserve-65bc85fbc-tcdmn"
Oct 30 22:35:55.051317 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:55.051217    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cccc800e-68e3-40ff-b7a0-29f5a777e83b-tmp-dir\") pod \"test-model-6922d3-kserve-65bc85fbc-tcdmn\" (UID: \"cccc800e-68e3-40ff-b7a0-29f5a777e83b\") " pod="ai-tenant-e2e-inf-a-6922d3/test-model-6922d3-kserve-65bc85fbc-tcdmn"
Oct 30 22:35:55.051535 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:55.051418    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cccc800e-68e3-40ff-b7a0-29f5a777e83b-home\") pod \"test-model-6922d3-kserve-65bc85fbc-tcdmn\" (UID: \"cccc800e-68e3-40ff-b7a0-29f5a777e83b\") " pod="ai-tenant-e2e-inf-a-6922d3/test-model-6922d3-kserve-65bc85fbc-tcdmn"
Oct 30 22:35:55.051535 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:55.051449    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cccc800e-68e3-40ff-b7a0-29f5a777e83b-model-cache\") pod \"test-model-6922d3-kserve-65bc85fbc-tcdmn\" (UID: \"cccc800e-68e3-40ff-b7a0-29f5a777e83b\") " pod="ai-tenant-e2e-inf-a-6922d3/test-model-6922d3-kserve-65bc85fbc-tcdmn"
Oct 30 22:35:55.051896 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:55.051859    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cccc800e-68e3-40ff-b7a0-29f5a777e83b-model-cache\") pod \"test-model-6922d3-kserve-65bc85fbc-tcdmn\" (UID: \"cccc800e-68e3-40ff-b7a0-29f5a777e83b\") " pod="ai-tenant-e2e-inf-a-6922d3/test-model-6922d3-kserve-65bc85fbc-tcdmn"
Oct 30 22:35:55.052006 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:55.051892    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cccc800e-68e3-40ff-b7a0-29f5a777e83b-home\") pod \"test-model-6922d3-kserve-65bc85fbc-tcdmn\" (UID: \"cccc800e-68e3-40ff-b7a0-29f5a777e83b\") " pod="ai-tenant-e2e-inf-a-6922d3/test-model-6922d3-kserve-65bc85fbc-tcdmn"
Oct 30 22:35:55.052254 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:55.052230    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cccc800e-68e3-40ff-b7a0-29f5a777e83b-tmp-dir\") pod \"test-model-6922d3-kserve-65bc85fbc-tcdmn\" (UID: \"cccc800e-68e3-40ff-b7a0-29f5a777e83b\") " pod="ai-tenant-e2e-inf-a-6922d3/test-model-6922d3-kserve-65bc85fbc-tcdmn"
Oct 30 22:35:55.053655 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:55.053606    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cccc800e-68e3-40ff-b7a0-29f5a777e83b-dshm\") pod \"test-model-6922d3-kserve-65bc85fbc-tcdmn\" (UID: \"cccc800e-68e3-40ff-b7a0-29f5a777e83b\") " pod="ai-tenant-e2e-inf-a-6922d3/test-model-6922d3-kserve-65bc85fbc-tcdmn"
Oct 30 22:35:55.053937 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:55.053914    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cccc800e-68e3-40ff-b7a0-29f5a777e83b-tls-certs\") pod \"test-model-6922d3-kserve-65bc85fbc-tcdmn\" (UID: \"cccc800e-68e3-40ff-b7a0-29f5a777e83b\") " pod="ai-tenant-e2e-inf-a-6922d3/test-model-6922d3-kserve-65bc85fbc-tcdmn"
Oct 30 22:35:55.068920 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:55.068899    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5vrx\" (UniqueName: \"kubernetes.io/projected/cccc800e-68e3-40ff-b7a0-29f5a777e83b-kube-api-access-l5vrx\") pod \"test-model-6922d3-kserve-65bc85fbc-tcdmn\" (UID: \"cccc800e-68e3-40ff-b7a0-29f5a777e83b\") " pod="ai-tenant-e2e-inf-a-6922d3/test-model-6922d3-kserve-65bc85fbc-tcdmn"
Oct 30 22:35:55.167088 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:55.164679    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-6922d3/test-model-6922d3-kserve-65bc85fbc-tcdmn"
Oct 30 22:35:55.553352 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:55.553327    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-6922d3/test-model-6922d3-kserve-65bc85fbc-tcdmn"]
Oct 30 22:35:55.556985 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:35:55.556953    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcccc800e_68e3_40ff_b7a0_29f5a777e83b.slice/crio-ff7385b77733623a6cce6363ae776f4884835d53cade3076d52ebb19ca5360a5 WatchSource:0}: Error finding container ff7385b77733623a6cce6363ae776f4884835d53cade3076d52ebb19ca5360a5: Status 404 returned error can't find the container with id ff7385b77733623a6cce6363ae776f4884835d53cade3076d52ebb19ca5360a5
Oct 30 22:35:56.268362 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:35:56.268318    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 22:35:56.329440 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:56.329405    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-6922d3/test-model-6922d3-kserve-65bc85fbc-tcdmn" event={"ID":"cccc800e-68e3-40ff-b7a0-29f5a777e83b","Type":"ContainerStarted","Data":"4496852a9575c119abd9627773fbc2798fd3ebe6417658082676189632951414"}
Oct 30 22:35:56.329654 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:56.329455    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-6922d3/test-model-6922d3-kserve-65bc85fbc-tcdmn" event={"ID":"cccc800e-68e3-40ff-b7a0-29f5a777e83b","Type":"ContainerStarted","Data":"ff7385b77733623a6cce6363ae776f4884835d53cade3076d52ebb19ca5360a5"}
Oct 30 22:35:56.367787 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:56.367700    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-6922d3/test-model-6922d3-kserve-65bc85fbc-tcdmn" podStartSLOduration=2.3676880000000002 podStartE2EDuration="2.367688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:35:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:35:56.365719423 +0000 UTC m=+1740.291645632" watchObservedRunningTime="2026-10-30 22:35:56.367688 +0000 UTC m=+1740.293614205"
Oct 30 22:35:57.165419 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:35:57.165386    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-6922d3/test-model-6922d3-kserve-65bc85fbc-tcdmn"
Oct 30 22:36:02.627138 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:02.627095    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8b3e43-68bdb79f49-9kn25"]
Oct 30 22:36:02.627613 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:02.627296    4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8b3e43-68bdb79f49-9kn25" podUID="0b547204-65d4-42e4-83c3-914a7ee0e951" containerName="maas-api" containerID="cri-o://faa9afe80a2de4322ee86d2bb6bd90cdf94cc4765234eb09174ebd4d2895294b" gracePeriod=30
Oct 30 22:36:03.062635 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:03.062410    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8b3e43-68bdb79f49-9kn25"
Oct 30 22:36:03.199397 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:03.199360    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b547204-65d4-42e4-83c3-914a7ee0e951-kube-api-access-vlxw4\" (UniqueName: \"kubernetes.io/projected/0b547204-65d4-42e4-83c3-914a7ee0e951-kube-api-access-vlxw4\") pod \"0b547204-65d4-42e4-83c3-914a7ee0e951\" (UID: \"0b547204-65d4-42e4-83c3-914a7ee0e951\") "
Oct 30 22:36:03.199648 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:03.199417    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b547204-65d4-42e4-83c3-914a7ee0e951-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b547204-65d4-42e4-83c3-914a7ee0e951-openshift-service-ca\") pod \"0b547204-65d4-42e4-83c3-914a7ee0e951\" (UID: \"0b547204-65d4-42e4-83c3-914a7ee0e951\") "
Oct 30 22:36:03.199648 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:03.199457    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b547204-65d4-42e4-83c3-914a7ee0e951-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b547204-65d4-42e4-83c3-914a7ee0e951-metrics-certs\") pod \"0b547204-65d4-42e4-83c3-914a7ee0e951\" (UID: \"0b547204-65d4-42e4-83c3-914a7ee0e951\") "
Oct 30 22:36:03.199648 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:03.199579    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b547204-65d4-42e4-83c3-914a7ee0e951-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b547204-65d4-42e4-83c3-914a7ee0e951-maas-api-tls\") pod \"0b547204-65d4-42e4-83c3-914a7ee0e951\" (UID: \"0b547204-65d4-42e4-83c3-914a7ee0e951\") "
Oct 30 22:36:03.200099 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:03.200063    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b547204-65d4-42e4-83c3-914a7ee0e951-openshift-service-ca" pod "0b547204-65d4-42e4-83c3-914a7ee0e951" (UID: "0b547204-65d4-42e4-83c3-914a7ee0e951"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:36:03.201806 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:03.201776    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b547204-65d4-42e4-83c3-914a7ee0e951-metrics-certs" pod "0b547204-65d4-42e4-83c3-914a7ee0e951" (UID: "0b547204-65d4-42e4-83c3-914a7ee0e951"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:36:03.201917 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:03.201856    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b547204-65d4-42e4-83c3-914a7ee0e951-maas-api-tls" pod "0b547204-65d4-42e4-83c3-914a7ee0e951" (UID: "0b547204-65d4-42e4-83c3-914a7ee0e951"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:36:03.201917 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:03.201887    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b547204-65d4-42e4-83c3-914a7ee0e951-kube-api-access-vlxw4" pod "0b547204-65d4-42e4-83c3-914a7ee0e951" (UID: "0b547204-65d4-42e4-83c3-914a7ee0e951"). InnerVolumeSpecName "kube-api-access-vlxw4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:36:03.302017 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:03.301930    4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlxw4\" (UniqueName: \"kubernetes.io/projected/0b547204-65d4-42e4-83c3-914a7ee0e951-kube-api-access-vlxw4\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:36:03.302017 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:03.301961    4477 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0b547204-65d4-42e4-83c3-914a7ee0e951-openshift-service-ca\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:36:03.302017 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:03.301972    4477 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0b547204-65d4-42e4-83c3-914a7ee0e951-metrics-certs\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:36:03.302017 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:03.301981    4477 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0b547204-65d4-42e4-83c3-914a7ee0e951-maas-api-tls\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:36:03.376157 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:03.376117    4477 generic.go:356] "Generic (PLEG): container finished" podID="0b547204-65d4-42e4-83c3-914a7ee0e951" containerID="faa9afe80a2de4322ee86d2bb6bd90cdf94cc4765234eb09174ebd4d2895294b" exitCode=0
Oct 30 22:36:03.376310 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:03.376171    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8b3e43-68bdb79f49-9kn25"
Oct 30 22:36:03.376310 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:03.376191    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8b3e43-68bdb79f49-9kn25" event={"ID":"0b547204-65d4-42e4-83c3-914a7ee0e951","Type":"ContainerDied","Data":"faa9afe80a2de4322ee86d2bb6bd90cdf94cc4765234eb09174ebd4d2895294b"}
Oct 30 22:36:03.376310 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:03.376253    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-8b3e43-68bdb79f49-9kn25" event={"ID":"0b547204-65d4-42e4-83c3-914a7ee0e951","Type":"ContainerDied","Data":"e5ce2a38f4858205560de528200c9339ddfd5a1058c0bc11b3a37451d70a6779"}
Oct 30 22:36:03.376310 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:03.376277    4477 scope.go:122] "RemoveContainer" containerID="faa9afe80a2de4322ee86d2bb6bd90cdf94cc4765234eb09174ebd4d2895294b"
Oct 30 22:36:03.400254 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:03.400231    4477 scope.go:122] "RemoveContainer" containerID="faa9afe80a2de4322ee86d2bb6bd90cdf94cc4765234eb09174ebd4d2895294b"
Oct 30 22:36:03.400544 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:36:03.400522    4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"faa9afe80a2de4322ee86d2bb6bd90cdf94cc4765234eb09174ebd4d2895294b\": container with ID starting with faa9afe80a2de4322ee86d2bb6bd90cdf94cc4765234eb09174ebd4d2895294b not found: ID does not exist" containerID="faa9afe80a2de4322ee86d2bb6bd90cdf94cc4765234eb09174ebd4d2895294b"
Oct 30 22:36:03.400630 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:03.400551    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"faa9afe80a2de4322ee86d2bb6bd90cdf94cc4765234eb09174ebd4d2895294b"} err="failed to get container status \"faa9afe80a2de4322ee86d2bb6bd90cdf94cc4765234eb09174ebd4d2895294b\": rpc error: code = NotFound desc = could not find container \"faa9afe80a2de4322ee86d2bb6bd90cdf94cc4765234eb09174ebd4d2895294b\": container with ID starting with faa9afe80a2de4322ee86d2bb6bd90cdf94cc4765234eb09174ebd4d2895294b not found: ID does not exist"
Oct 30 22:36:03.423743 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:03.423723    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8b3e43-68bdb79f49-9kn25"]
Oct 30 22:36:03.427097 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:03.427066    4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-8b3e43-68bdb79f49-9kn25"]
Oct 30 22:36:04.194847 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:04.194814    4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b547204-65d4-42e4-83c3-914a7ee0e951" path="/var/lib/kubelet/pods/0b547204-65d4-42e4-83c3-914a7ee0e951/volumes"
Oct 30 22:36:05.165487 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:05.165451    4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-6922d3/test-model-6922d3-kserve-65bc85fbc-tcdmn"
Oct 30 22:36:05.178444 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:05.178419    4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-6922d3/test-model-6922d3-kserve-65bc85fbc-tcdmn"
Oct 30 22:36:05.401655 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:05.401627    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-6922d3/test-model-6922d3-kserve-65bc85fbc-tcdmn"
Oct 30 22:36:05.647107 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:05.647075    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-999470-5dcc669f68-nhc2b"]
Oct 30 22:36:05.647699 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:05.647682    4477 cpu_manager.go:414] "containerMap: removing container" podUID="0b547204-65d4-42e4-83c3-914a7ee0e951" containerName="maas-api"
Oct 30 22:36:05.647699 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:05.647698    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b547204-65d4-42e4-83c3-914a7ee0e951" containerName="maas-api"
Oct 30 22:36:05.647855 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:05.647830    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b547204-65d4-42e4-83c3-914a7ee0e951" containerName="maas-api"
Oct 30 22:36:05.652847 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:05.652825    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-999470-5dcc669f68-nhc2b"
Oct 30 22:36:05.667848 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:05.667827    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-999470-serving-cert\""
Oct 30 22:36:05.680628 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:05.680595    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-999470-5dcc669f68-nhc2b"]
Oct 30 22:36:05.732798 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:05.732768    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/09adb84c-9a9c-466e-8e9e-8b53437593af-openshift-service-ca\") pod \"maas-api-e2e-inf-a-999470-5dcc669f68-nhc2b\" (UID: \"09adb84c-9a9c-466e-8e9e-8b53437593af\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-999470-5dcc669f68-nhc2b"
Oct 30 22:36:05.732960 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:05.732821    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/09adb84c-9a9c-466e-8e9e-8b53437593af-maas-api-tls\") pod \"maas-api-e2e-inf-a-999470-5dcc669f68-nhc2b\" (UID: \"09adb84c-9a9c-466e-8e9e-8b53437593af\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-999470-5dcc669f68-nhc2b"
Oct 30 22:36:05.732960 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:05.732867    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/09adb84c-9a9c-466e-8e9e-8b53437593af-metrics-certs\") pod \"maas-api-e2e-inf-a-999470-5dcc669f68-nhc2b\" (UID: \"09adb84c-9a9c-466e-8e9e-8b53437593af\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-999470-5dcc669f68-nhc2b"
Oct 30 22:36:05.733151 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:05.733126    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jgfg\" (UniqueName: \"kubernetes.io/projected/09adb84c-9a9c-466e-8e9e-8b53437593af-kube-api-access-4jgfg\") pod \"maas-api-e2e-inf-a-999470-5dcc669f68-nhc2b\" (UID: \"09adb84c-9a9c-466e-8e9e-8b53437593af\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-999470-5dcc669f68-nhc2b"
Oct 30 22:36:05.834429 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:05.834345    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jgfg\" (UniqueName: \"kubernetes.io/projected/09adb84c-9a9c-466e-8e9e-8b53437593af-kube-api-access-4jgfg\") pod \"maas-api-e2e-inf-a-999470-5dcc669f68-nhc2b\" (UID: \"09adb84c-9a9c-466e-8e9e-8b53437593af\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-999470-5dcc669f68-nhc2b"
Oct 30 22:36:05.834429 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:05.834435    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/09adb84c-9a9c-466e-8e9e-8b53437593af-openshift-service-ca\") pod \"maas-api-e2e-inf-a-999470-5dcc669f68-nhc2b\" (UID: \"09adb84c-9a9c-466e-8e9e-8b53437593af\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-999470-5dcc669f68-nhc2b"
Oct 30 22:36:05.834878 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:05.834471    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/09adb84c-9a9c-466e-8e9e-8b53437593af-maas-api-tls\") pod \"maas-api-e2e-inf-a-999470-5dcc669f68-nhc2b\" (UID: \"09adb84c-9a9c-466e-8e9e-8b53437593af\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-999470-5dcc669f68-nhc2b"
Oct 30 22:36:05.834878 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:05.834568    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/09adb84c-9a9c-466e-8e9e-8b53437593af-metrics-certs\") pod \"maas-api-e2e-inf-a-999470-5dcc669f68-nhc2b\" (UID: \"09adb84c-9a9c-466e-8e9e-8b53437593af\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-999470-5dcc669f68-nhc2b"
Oct 30 22:36:05.834878 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:36:05.834583    4477 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-999470-serving-cert: secret "maas-api-e2e-inf-a-999470-serving-cert" not found
Oct 30 22:36:05.834878 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:36:05.834658    4477 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/09adb84c-9a9c-466e-8e9e-8b53437593af-maas-api-tls podName:09adb84c-9a9c-466e-8e9e-8b53437593af nodeName:}" failed. No retries permitted until 2026-10-30 22:36:06.334635869 +0000 UTC m=+1750.260562073 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/09adb84c-9a9c-466e-8e9e-8b53437593af-maas-api-tls") pod "maas-api-e2e-inf-a-999470-5dcc669f68-nhc2b" (UID: "09adb84c-9a9c-466e-8e9e-8b53437593af") : secret "maas-api-e2e-inf-a-999470-serving-cert" not found
Oct 30 22:36:05.835233 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:05.835212    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/09adb84c-9a9c-466e-8e9e-8b53437593af-openshift-service-ca\") pod \"maas-api-e2e-inf-a-999470-5dcc669f68-nhc2b\" (UID: \"09adb84c-9a9c-466e-8e9e-8b53437593af\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-999470-5dcc669f68-nhc2b"
Oct 30 22:36:05.836987 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:05.836970    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/09adb84c-9a9c-466e-8e9e-8b53437593af-metrics-certs\") pod \"maas-api-e2e-inf-a-999470-5dcc669f68-nhc2b\" (UID: \"09adb84c-9a9c-466e-8e9e-8b53437593af\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-999470-5dcc669f68-nhc2b"
Oct 30 22:36:05.854222 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:05.854188    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jgfg\" (UniqueName: \"kubernetes.io/projected/09adb84c-9a9c-466e-8e9e-8b53437593af-kube-api-access-4jgfg\") pod \"maas-api-e2e-inf-a-999470-5dcc669f68-nhc2b\" (UID: \"09adb84c-9a9c-466e-8e9e-8b53437593af\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-999470-5dcc669f68-nhc2b"
Oct 30 22:36:06.342303 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:06.342269    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/09adb84c-9a9c-466e-8e9e-8b53437593af-maas-api-tls\") pod \"maas-api-e2e-inf-a-999470-5dcc669f68-nhc2b\" (UID: \"09adb84c-9a9c-466e-8e9e-8b53437593af\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-999470-5dcc669f68-nhc2b"
Oct 30 22:36:06.344683 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:06.344658    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/09adb84c-9a9c-466e-8e9e-8b53437593af-maas-api-tls\") pod \"maas-api-e2e-inf-a-999470-5dcc669f68-nhc2b\" (UID: \"09adb84c-9a9c-466e-8e9e-8b53437593af\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-999470-5dcc669f68-nhc2b"
Oct 30 22:36:06.567929 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:06.567899    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-999470-5dcc669f68-nhc2b"
Oct 30 22:36:06.883227 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:06.883198    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-999470-5dcc669f68-nhc2b"]
Oct 30 22:36:06.888449 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:36:06.888388    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09adb84c_9a9c_466e_8e9e_8b53437593af.slice/crio-18e208e6113b9d0e62b84b833fec64838eb27bab154c962f15d2f72d17d4d12d WatchSource:0}: Error finding container 18e208e6113b9d0e62b84b833fec64838eb27bab154c962f15d2f72d17d4d12d: Status 404 returned error can't find the container with id 18e208e6113b9d0e62b84b833fec64838eb27bab154c962f15d2f72d17d4d12d
Oct 30 22:36:06.889757 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:06.889736    4477 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 22:36:07.405314 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:07.405263    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-999470-5dcc669f68-nhc2b" event={"ID":"09adb84c-9a9c-466e-8e9e-8b53437593af","Type":"ContainerStarted","Data":"b9cff0ec35bcac86e7d049db5988b0c54ee1184ad7f906bf6cd54cb5eae4208c"}
Oct 30 22:36:07.405477 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:07.405326    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-999470-5dcc669f68-nhc2b" event={"ID":"09adb84c-9a9c-466e-8e9e-8b53437593af","Type":"ContainerStarted","Data":"18e208e6113b9d0e62b84b833fec64838eb27bab154c962f15d2f72d17d4d12d"}
Oct 30 22:36:07.405546 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:07.405533    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-999470-5dcc669f68-nhc2b"
Oct 30 22:36:07.443072 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:07.442992    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-999470-5dcc669f68-nhc2b" podStartSLOduration=2.090872578 podStartE2EDuration="2.442981074s" totalImagesPullingTime="352.108496ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:36:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:36:07.442233448 +0000 UTC m=+1751.368159651" watchObservedRunningTime="2026-10-30 22:36:07.442981074 +0000 UTC m=+1751.368907311"
Oct 30 22:36:10.440829 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:10.440680    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-m74tl"]
Oct 30 22:36:10.441295 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:10.440915    4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-m74tl" podUID="3e6f6850-1ca2-4f4c-a047-23e35ce2b697" containerName="istio-proxy" containerID="cri-o://ae78dc236996f12012a82d398789cf29f4474cf4a6bdc08ebf463df6b98553b3" gracePeriod=30
Oct 30 22:36:13.418614 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:13.418581    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-999470-5dcc669f68-nhc2b"
Oct 30 22:36:15.725320 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:15.725254    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-m74tl"
Oct 30 22:36:15.789581 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:15.789545    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e6f6850-1ca2-4f4c-a047-23e35ce2b697-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e6f6850-1ca2-4f4c-a047-23e35ce2b697-istio-data\") pod \"3e6f6850-1ca2-4f4c-a047-23e35ce2b697\" (UID: \"3e6f6850-1ca2-4f4c-a047-23e35ce2b697\") "
Oct 30 22:36:15.789754 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:15.789593    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e6f6850-1ca2-4f4c-a047-23e35ce2b697-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e6f6850-1ca2-4f4c-a047-23e35ce2b697-workload-certs\") pod \"3e6f6850-1ca2-4f4c-a047-23e35ce2b697\" (UID: \"3e6f6850-1ca2-4f4c-a047-23e35ce2b697\") "
Oct 30 22:36:15.789831 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:15.789770    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e6f6850-1ca2-4f4c-a047-23e35ce2b697-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e6f6850-1ca2-4f4c-a047-23e35ce2b697-istio-envoy\") pod \"3e6f6850-1ca2-4f4c-a047-23e35ce2b697\" (UID: \"3e6f6850-1ca2-4f4c-a047-23e35ce2b697\") "
Oct 30 22:36:15.789889 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:15.789868    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3e6f6850-1ca2-4f4c-a047-23e35ce2b697-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e6f6850-1ca2-4f4c-a047-23e35ce2b697-istio-podinfo\") pod \"3e6f6850-1ca2-4f4c-a047-23e35ce2b697\" (UID: \"3e6f6850-1ca2-4f4c-a047-23e35ce2b697\") "
Oct 30 22:36:15.789946 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:15.789890    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e6f6850-1ca2-4f4c-a047-23e35ce2b697-istio-data" pod "3e6f6850-1ca2-4f4c-a047-23e35ce2b697" (UID: "3e6f6850-1ca2-4f4c-a047-23e35ce2b697"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:15.789946 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:15.789909    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e6f6850-1ca2-4f4c-a047-23e35ce2b697-kube-api-access-6nfhf\" (UniqueName: \"kubernetes.io/projected/3e6f6850-1ca2-4f4c-a047-23e35ce2b697-kube-api-access-6nfhf\") pod \"3e6f6850-1ca2-4f4c-a047-23e35ce2b697\" (UID: \"3e6f6850-1ca2-4f4c-a047-23e35ce2b697\") "
Oct 30 22:36:15.790094 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:15.789940    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e6f6850-1ca2-4f4c-a047-23e35ce2b697-workload-certs" pod "3e6f6850-1ca2-4f4c-a047-23e35ce2b697" (UID: "3e6f6850-1ca2-4f4c-a047-23e35ce2b697"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:15.790094 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:15.789972    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e6f6850-1ca2-4f4c-a047-23e35ce2b697-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e6f6850-1ca2-4f4c-a047-23e35ce2b697-workload-socket\") pod \"3e6f6850-1ca2-4f4c-a047-23e35ce2b697\" (UID: \"3e6f6850-1ca2-4f4c-a047-23e35ce2b697\") "
Oct 30 22:36:15.790094 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:15.790011    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e6f6850-1ca2-4f4c-a047-23e35ce2b697-istio-token\" (UniqueName: \"kubernetes.io/projected/3e6f6850-1ca2-4f4c-a047-23e35ce2b697-istio-token\") pod \"3e6f6850-1ca2-4f4c-a047-23e35ce2b697\" (UID: \"3e6f6850-1ca2-4f4c-a047-23e35ce2b697\") "
Oct 30 22:36:15.790094 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:15.790069    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e6f6850-1ca2-4f4c-a047-23e35ce2b697-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e6f6850-1ca2-4f4c-a047-23e35ce2b697-credential-socket\") pod \"3e6f6850-1ca2-4f4c-a047-23e35ce2b697\" (UID: \"3e6f6850-1ca2-4f4c-a047-23e35ce2b697\") "
Oct 30 22:36:15.790317 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:15.790168    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e6f6850-1ca2-4f4c-a047-23e35ce2b697-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e6f6850-1ca2-4f4c-a047-23e35ce2b697-istiod-ca-cert\") pod \"3e6f6850-1ca2-4f4c-a047-23e35ce2b697\" (UID: \"3e6f6850-1ca2-4f4c-a047-23e35ce2b697\") "
Oct 30 22:36:15.790429 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:15.790403    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e6f6850-1ca2-4f4c-a047-23e35ce2b697-workload-socket" pod "3e6f6850-1ca2-4f4c-a047-23e35ce2b697" (UID: "3e6f6850-1ca2-4f4c-a047-23e35ce2b697"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:15.791228 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:15.790898    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e6f6850-1ca2-4f4c-a047-23e35ce2b697-credential-socket" pod "3e6f6850-1ca2-4f4c-a047-23e35ce2b697" (UID: "3e6f6850-1ca2-4f4c-a047-23e35ce2b697"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:15.791228 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:15.790912    4477 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e6f6850-1ca2-4f4c-a047-23e35ce2b697-istio-data\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:36:15.791228 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:15.790952    4477 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e6f6850-1ca2-4f4c-a047-23e35ce2b697-workload-certs\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:36:15.791228 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:15.790968    4477 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e6f6850-1ca2-4f4c-a047-23e35ce2b697-workload-socket\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:36:15.791228 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:15.791081    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e6f6850-1ca2-4f4c-a047-23e35ce2b697-istiod-ca-cert" pod "3e6f6850-1ca2-4f4c-a047-23e35ce2b697" (UID: "3e6f6850-1ca2-4f4c-a047-23e35ce2b697"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:36:15.793147 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:15.793124    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3e6f6850-1ca2-4f4c-a047-23e35ce2b697-istio-podinfo" pod "3e6f6850-1ca2-4f4c-a047-23e35ce2b697" (UID: "3e6f6850-1ca2-4f4c-a047-23e35ce2b697"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:36:15.808669 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:15.808617    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e6f6850-1ca2-4f4c-a047-23e35ce2b697-istio-envoy" pod "3e6f6850-1ca2-4f4c-a047-23e35ce2b697" (UID: "3e6f6850-1ca2-4f4c-a047-23e35ce2b697"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:15.808669 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:15.808657    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e6f6850-1ca2-4f4c-a047-23e35ce2b697-istio-token" pod "3e6f6850-1ca2-4f4c-a047-23e35ce2b697" (UID: "3e6f6850-1ca2-4f4c-a047-23e35ce2b697"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:36:15.808870 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:15.808746    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e6f6850-1ca2-4f4c-a047-23e35ce2b697-kube-api-access-6nfhf" pod "3e6f6850-1ca2-4f4c-a047-23e35ce2b697" (UID: "3e6f6850-1ca2-4f4c-a047-23e35ce2b697"). InnerVolumeSpecName "kube-api-access-6nfhf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:36:15.894602 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:15.894570    4477 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e6f6850-1ca2-4f4c-a047-23e35ce2b697-istio-envoy\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:36:15.894602 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:15.894601    4477 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e6f6850-1ca2-4f4c-a047-23e35ce2b697-istio-podinfo\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:36:15.894794 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:15.894610    4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nfhf\" (UniqueName: \"kubernetes.io/projected/3e6f6850-1ca2-4f4c-a047-23e35ce2b697-kube-api-access-6nfhf\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:36:15.894794 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:15.894619    4477 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e6f6850-1ca2-4f4c-a047-23e35ce2b697-istio-token\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:36:15.894794 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:15.894630    4477 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e6f6850-1ca2-4f4c-a047-23e35ce2b697-credential-socket\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:36:15.894794 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:15.894638    4477 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e6f6850-1ca2-4f4c-a047-23e35ce2b697-istiod-ca-cert\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:36:16.484219 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:16.484112    4477 generic.go:356] "Generic (PLEG): container finished" podID="3e6f6850-1ca2-4f4c-a047-23e35ce2b697" containerID="ae78dc236996f12012a82d398789cf29f4474cf4a6bdc08ebf463df6b98553b3" exitCode=0
Oct 30 22:36:16.484626 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:16.484585    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-m74tl" event={"ID":"3e6f6850-1ca2-4f4c-a047-23e35ce2b697","Type":"ContainerDied","Data":"ae78dc236996f12012a82d398789cf29f4474cf4a6bdc08ebf463df6b98553b3"}
Oct 30 22:36:16.484773 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:16.484673    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-m74tl" event={"ID":"3e6f6850-1ca2-4f4c-a047-23e35ce2b697","Type":"ContainerDied","Data":"c008e4e77dd4e1d366bd28ddacdc81f83fd848320fb072983ab756d59d15f9ba"}
Oct 30 22:36:16.484773 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:16.484701    4477 scope.go:122] "RemoveContainer" containerID="ae78dc236996f12012a82d398789cf29f4474cf4a6bdc08ebf463df6b98553b3"
Oct 30 22:36:16.484882 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:16.484871    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-m74tl"
Oct 30 22:36:16.505361 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:16.505333    4477 scope.go:122] "RemoveContainer" containerID="ae78dc236996f12012a82d398789cf29f4474cf4a6bdc08ebf463df6b98553b3"
Oct 30 22:36:16.505789 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:36:16.505763    4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae78dc236996f12012a82d398789cf29f4474cf4a6bdc08ebf463df6b98553b3\": container with ID starting with ae78dc236996f12012a82d398789cf29f4474cf4a6bdc08ebf463df6b98553b3 not found: ID does not exist" containerID="ae78dc236996f12012a82d398789cf29f4474cf4a6bdc08ebf463df6b98553b3"
Oct 30 22:36:16.505869 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:16.505802    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae78dc236996f12012a82d398789cf29f4474cf4a6bdc08ebf463df6b98553b3"} err="failed to get container status \"ae78dc236996f12012a82d398789cf29f4474cf4a6bdc08ebf463df6b98553b3\": rpc error: code = NotFound desc = could not find container \"ae78dc236996f12012a82d398789cf29f4474cf4a6bdc08ebf463df6b98553b3\": container with ID starting with ae78dc236996f12012a82d398789cf29f4474cf4a6bdc08ebf463df6b98553b3 not found: ID does not exist"
Oct 30 22:36:16.545310 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:16.545266    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-m74tl"]
Oct 30 22:36:16.555215 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:16.555179    4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-m74tl"]
Oct 30 22:36:18.194897 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:18.194862    4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e6f6850-1ca2-4f4c-a047-23e35ce2b697" path="/var/lib/kubelet/pods/3e6f6850-1ca2-4f4c-a047-23e35ce2b697/volumes"
Oct 30 22:36:24.967998 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:24.967965    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-f9d87b/test-model-f9d87b-kserve-58b9dc6999-ttktm"]
Oct 30 22:36:24.968900 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:24.968876    4477 cpu_manager.go:414] "containerMap: removing container" podUID="3e6f6850-1ca2-4f4c-a047-23e35ce2b697" containerName="istio-proxy"
Oct 30 22:36:24.968900 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:24.968899    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e6f6850-1ca2-4f4c-a047-23e35ce2b697" containerName="istio-proxy"
Oct 30 22:36:24.969134 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:24.969056    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e6f6850-1ca2-4f4c-a047-23e35ce2b697" containerName="istio-proxy"
Oct 30 22:36:24.972661 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:24.972639    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f9d87b/test-model-f9d87b-kserve-58b9dc6999-ttktm"
Oct 30 22:36:25.002366 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:25.002338    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-f9d87b\"/\"test-model-f9d87b-kserve-self-signed-certs\""
Oct 30 22:36:25.002532 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:25.002447    4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-f9d87b\"/\"kube-root-ca.crt\""
Oct 30 22:36:25.002532 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:25.002537    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-f9d87b\"/\"default-dockercfg-cvmpd\""
Oct 30 22:36:25.002532 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:25.002605    4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-f9d87b\"/\"openshift-service-ca.crt\""
Oct 30 22:36:25.007529 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:25.007500    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-f9d87b/test-model-f9d87b-kserve-58b9dc6999-ttktm"]
Oct 30 22:36:25.068403 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:25.068371    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e7cf3ef-0cd6-415f-8fb5-b806fd818c77-dshm\") pod \"test-model-f9d87b-kserve-58b9dc6999-ttktm\" (UID: \"5e7cf3ef-0cd6-415f-8fb5-b806fd818c77\") " pod="ai-tenant-e2e-inf-b-f9d87b/test-model-f9d87b-kserve-58b9dc6999-ttktm"
Oct 30 22:36:25.068561 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:25.068514    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e7cf3ef-0cd6-415f-8fb5-b806fd818c77-home\") pod \"test-model-f9d87b-kserve-58b9dc6999-ttktm\" (UID: \"5e7cf3ef-0cd6-415f-8fb5-b806fd818c77\") " pod="ai-tenant-e2e-inf-b-f9d87b/test-model-f9d87b-kserve-58b9dc6999-ttktm"
Oct 30 22:36:25.068626 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:25.068583    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2vbl\" (UniqueName: \"kubernetes.io/projected/5e7cf3ef-0cd6-415f-8fb5-b806fd818c77-kube-api-access-q2vbl\") pod \"test-model-f9d87b-kserve-58b9dc6999-ttktm\" (UID: \"5e7cf3ef-0cd6-415f-8fb5-b806fd818c77\") " pod="ai-tenant-e2e-inf-b-f9d87b/test-model-f9d87b-kserve-58b9dc6999-ttktm"
Oct 30 22:36:25.068675 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:25.068658    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e7cf3ef-0cd6-415f-8fb5-b806fd818c77-tls-certs\") pod \"test-model-f9d87b-kserve-58b9dc6999-ttktm\" (UID: \"5e7cf3ef-0cd6-415f-8fb5-b806fd818c77\") " pod="ai-tenant-e2e-inf-b-f9d87b/test-model-f9d87b-kserve-58b9dc6999-ttktm"
Oct 30 22:36:25.068822 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:25.068796    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e7cf3ef-0cd6-415f-8fb5-b806fd818c77-model-cache\") pod \"test-model-f9d87b-kserve-58b9dc6999-ttktm\" (UID: \"5e7cf3ef-0cd6-415f-8fb5-b806fd818c77\") " pod="ai-tenant-e2e-inf-b-f9d87b/test-model-f9d87b-kserve-58b9dc6999-ttktm"
Oct 30 22:36:25.068891 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:25.068854    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e7cf3ef-0cd6-415f-8fb5-b806fd818c77-tmp-dir\") pod \"test-model-f9d87b-kserve-58b9dc6999-ttktm\" (UID: \"5e7cf3ef-0cd6-415f-8fb5-b806fd818c77\") " pod="ai-tenant-e2e-inf-b-f9d87b/test-model-f9d87b-kserve-58b9dc6999-ttktm"
Oct 30 22:36:25.169734 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:25.169704    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e7cf3ef-0cd6-415f-8fb5-b806fd818c77-model-cache\") pod \"test-model-f9d87b-kserve-58b9dc6999-ttktm\" (UID: \"5e7cf3ef-0cd6-415f-8fb5-b806fd818c77\") " pod="ai-tenant-e2e-inf-b-f9d87b/test-model-f9d87b-kserve-58b9dc6999-ttktm"
Oct 30 22:36:25.169883 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:25.169751    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e7cf3ef-0cd6-415f-8fb5-b806fd818c77-tmp-dir\") pod \"test-model-f9d87b-kserve-58b9dc6999-ttktm\" (UID: \"5e7cf3ef-0cd6-415f-8fb5-b806fd818c77\") " pod="ai-tenant-e2e-inf-b-f9d87b/test-model-f9d87b-kserve-58b9dc6999-ttktm"
Oct 30 22:36:25.169961 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:25.169879    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e7cf3ef-0cd6-415f-8fb5-b806fd818c77-dshm\") pod \"test-model-f9d87b-kserve-58b9dc6999-ttktm\" (UID: \"5e7cf3ef-0cd6-415f-8fb5-b806fd818c77\") " pod="ai-tenant-e2e-inf-b-f9d87b/test-model-f9d87b-kserve-58b9dc6999-ttktm"
Oct 30 22:36:25.169961 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:25.169943    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e7cf3ef-0cd6-415f-8fb5-b806fd818c77-home\") pod \"test-model-f9d87b-kserve-58b9dc6999-ttktm\" (UID: \"5e7cf3ef-0cd6-415f-8fb5-b806fd818c77\") " pod="ai-tenant-e2e-inf-b-f9d87b/test-model-f9d87b-kserve-58b9dc6999-ttktm"
Oct 30 22:36:25.170124 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:25.169968    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2vbl\" (UniqueName: \"kubernetes.io/projected/5e7cf3ef-0cd6-415f-8fb5-b806fd818c77-kube-api-access-q2vbl\") pod \"test-model-f9d87b-kserve-58b9dc6999-ttktm\" (UID: \"5e7cf3ef-0cd6-415f-8fb5-b806fd818c77\") " pod="ai-tenant-e2e-inf-b-f9d87b/test-model-f9d87b-kserve-58b9dc6999-ttktm"
Oct 30 22:36:25.170124 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:25.169992    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e7cf3ef-0cd6-415f-8fb5-b806fd818c77-tls-certs\") pod \"test-model-f9d87b-kserve-58b9dc6999-ttktm\" (UID: \"5e7cf3ef-0cd6-415f-8fb5-b806fd818c77\") " pod="ai-tenant-e2e-inf-b-f9d87b/test-model-f9d87b-kserve-58b9dc6999-ttktm"
Oct 30 22:36:25.170234 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:25.170177    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e7cf3ef-0cd6-415f-8fb5-b806fd818c77-model-cache\") pod \"test-model-f9d87b-kserve-58b9dc6999-ttktm\" (UID: \"5e7cf3ef-0cd6-415f-8fb5-b806fd818c77\") " pod="ai-tenant-e2e-inf-b-f9d87b/test-model-f9d87b-kserve-58b9dc6999-ttktm"
Oct 30 22:36:25.170284 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:25.170259    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e7cf3ef-0cd6-415f-8fb5-b806fd818c77-tmp-dir\") pod \"test-model-f9d87b-kserve-58b9dc6999-ttktm\" (UID: \"5e7cf3ef-0cd6-415f-8fb5-b806fd818c77\") " pod="ai-tenant-e2e-inf-b-f9d87b/test-model-f9d87b-kserve-58b9dc6999-ttktm"
Oct 30 22:36:25.170443 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:25.170424    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e7cf3ef-0cd6-415f-8fb5-b806fd818c77-home\") pod \"test-model-f9d87b-kserve-58b9dc6999-ttktm\" (UID: \"5e7cf3ef-0cd6-415f-8fb5-b806fd818c77\") " pod="ai-tenant-e2e-inf-b-f9d87b/test-model-f9d87b-kserve-58b9dc6999-ttktm"
Oct 30 22:36:25.172890 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:25.172867    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e7cf3ef-0cd6-415f-8fb5-b806fd818c77-tls-certs\") pod \"test-model-f9d87b-kserve-58b9dc6999-ttktm\" (UID: \"5e7cf3ef-0cd6-415f-8fb5-b806fd818c77\") " pod="ai-tenant-e2e-inf-b-f9d87b/test-model-f9d87b-kserve-58b9dc6999-ttktm"
Oct 30 22:36:25.173222 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:25.173204    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e7cf3ef-0cd6-415f-8fb5-b806fd818c77-dshm\") pod \"test-model-f9d87b-kserve-58b9dc6999-ttktm\" (UID: \"5e7cf3ef-0cd6-415f-8fb5-b806fd818c77\") " pod="ai-tenant-e2e-inf-b-f9d87b/test-model-f9d87b-kserve-58b9dc6999-ttktm"
Oct 30 22:36:25.190360 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:25.190327    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2vbl\" (UniqueName: \"kubernetes.io/projected/5e7cf3ef-0cd6-415f-8fb5-b806fd818c77-kube-api-access-q2vbl\") pod \"test-model-f9d87b-kserve-58b9dc6999-ttktm\" (UID: \"5e7cf3ef-0cd6-415f-8fb5-b806fd818c77\") " pod="ai-tenant-e2e-inf-b-f9d87b/test-model-f9d87b-kserve-58b9dc6999-ttktm"
Oct 30 22:36:25.287515 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:25.287428    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f9d87b/test-model-f9d87b-kserve-58b9dc6999-ttktm"
Oct 30 22:36:25.689596 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:25.689571    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-f9d87b/test-model-f9d87b-kserve-58b9dc6999-ttktm"]
Oct 30 22:36:25.692560 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:36:25.692533    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5e7cf3ef_0cd6_415f_8fb5_b806fd818c77.slice/crio-d6a84291848de127dd2793fa303e0060ba8eb2b72cb7695c9345b3f5d71756cb WatchSource:0}: Error finding container d6a84291848de127dd2793fa303e0060ba8eb2b72cb7695c9345b3f5d71756cb: Status 404 returned error can't find the container with id d6a84291848de127dd2793fa303e0060ba8eb2b72cb7695c9345b3f5d71756cb
Oct 30 22:36:26.267722 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:36:26.267685    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 22:36:26.574839 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:26.574766    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f9d87b/test-model-f9d87b-kserve-58b9dc6999-ttktm" event={"ID":"5e7cf3ef-0cd6-415f-8fb5-b806fd818c77","Type":"ContainerStarted","Data":"626b03c860aae36166a7292849a84b26fc7acfc9429912380442dfa01b3f04f0"}
Oct 30 22:36:26.574839 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:26.574818    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f9d87b/test-model-f9d87b-kserve-58b9dc6999-ttktm" event={"ID":"5e7cf3ef-0cd6-415f-8fb5-b806fd818c77","Type":"ContainerStarted","Data":"d6a84291848de127dd2793fa303e0060ba8eb2b72cb7695c9345b3f5d71756cb"}
Oct 30 22:36:26.619994 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:26.619937    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-f9d87b/test-model-f9d87b-kserve-58b9dc6999-ttktm" podStartSLOduration=2.61991874 podStartE2EDuration="2.61991874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:36:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:36:26.616621817 +0000 UTC m=+1770.542548020" watchObservedRunningTime="2026-10-30 22:36:26.61991874 +0000 UTC m=+1770.545844948"
Oct 30 22:36:27.287973 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:27.287943    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-f9d87b/test-model-f9d87b-kserve-58b9dc6999-ttktm"
Oct 30 22:36:35.288107 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:35.288068    4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-f9d87b/test-model-f9d87b-kserve-58b9dc6999-ttktm"
Oct 30 22:36:35.300495 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:35.300472    4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-f9d87b/test-model-f9d87b-kserve-58b9dc6999-ttktm"
Oct 30 22:36:35.667219 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:35.667184    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-f9d87b/test-model-f9d87b-kserve-58b9dc6999-ttktm"
Oct 30 22:36:42.135259 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:42.135228    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4f7439-85d59f65c5-5j5r5"]
Oct 30 22:36:42.135723 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:42.135674    4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4f7439-85d59f65c5-5j5r5" podUID="36f8f834-bec7-411d-a4bc-86c141eecc14" containerName="maas-api" containerID="cri-o://0a591696b32e0c890267bf149eb77fc8887c4444f0c9dde3ec63f522bc8c5296" gracePeriod=30
Oct 30 22:36:42.459094 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:42.451127    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4f7439-85d59f65c5-5j5r5"
Oct 30 22:36:42.624895 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:42.624860    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36f8f834-bec7-411d-a4bc-86c141eecc14-kube-api-access-nw72p\" (UniqueName: \"kubernetes.io/projected/36f8f834-bec7-411d-a4bc-86c141eecc14-kube-api-access-nw72p\") pod \"36f8f834-bec7-411d-a4bc-86c141eecc14\" (UID: \"36f8f834-bec7-411d-a4bc-86c141eecc14\") "
Oct 30 22:36:42.625114 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:42.625093    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36f8f834-bec7-411d-a4bc-86c141eecc14-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36f8f834-bec7-411d-a4bc-86c141eecc14-maas-api-tls\") pod \"36f8f834-bec7-411d-a4bc-86c141eecc14\" (UID: \"36f8f834-bec7-411d-a4bc-86c141eecc14\") "
Oct 30 22:36:42.625195 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:42.625121    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36f8f834-bec7-411d-a4bc-86c141eecc14-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36f8f834-bec7-411d-a4bc-86c141eecc14-openshift-service-ca\") pod \"36f8f834-bec7-411d-a4bc-86c141eecc14\" (UID: \"36f8f834-bec7-411d-a4bc-86c141eecc14\") "
Oct 30 22:36:42.625195 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:42.625164    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36f8f834-bec7-411d-a4bc-86c141eecc14-metrics-certs\" (UniqueName: \"kubernetes.io/secret/36f8f834-bec7-411d-a4bc-86c141eecc14-metrics-certs\") pod \"36f8f834-bec7-411d-a4bc-86c141eecc14\" (UID: \"36f8f834-bec7-411d-a4bc-86c141eecc14\") "
Oct 30 22:36:42.628022 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:42.627976    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36f8f834-bec7-411d-a4bc-86c141eecc14-openshift-service-ca" pod "36f8f834-bec7-411d-a4bc-86c141eecc14" (UID: "36f8f834-bec7-411d-a4bc-86c141eecc14"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:36:42.628573 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:42.628303    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36f8f834-bec7-411d-a4bc-86c141eecc14-metrics-certs" pod "36f8f834-bec7-411d-a4bc-86c141eecc14" (UID: "36f8f834-bec7-411d-a4bc-86c141eecc14"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:36:42.628711 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:42.628624    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36f8f834-bec7-411d-a4bc-86c141eecc14-kube-api-access-nw72p" pod "36f8f834-bec7-411d-a4bc-86c141eecc14" (UID: "36f8f834-bec7-411d-a4bc-86c141eecc14"). InnerVolumeSpecName "kube-api-access-nw72p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:36:42.630811 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:42.630770    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36f8f834-bec7-411d-a4bc-86c141eecc14-maas-api-tls" pod "36f8f834-bec7-411d-a4bc-86c141eecc14" (UID: "36f8f834-bec7-411d-a4bc-86c141eecc14"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:36:42.708767 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:42.708625    4477 generic.go:356] "Generic (PLEG): container finished" podID="36f8f834-bec7-411d-a4bc-86c141eecc14" containerID="0a591696b32e0c890267bf149eb77fc8887c4444f0c9dde3ec63f522bc8c5296" exitCode=0
Oct 30 22:36:42.708767 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:42.708699    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4f7439-85d59f65c5-5j5r5"
Oct 30 22:36:42.708767 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:42.708745    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4f7439-85d59f65c5-5j5r5" event={"ID":"36f8f834-bec7-411d-a4bc-86c141eecc14","Type":"ContainerDied","Data":"0a591696b32e0c890267bf149eb77fc8887c4444f0c9dde3ec63f522bc8c5296"}
Oct 30 22:36:42.709272 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:42.708909    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4f7439-85d59f65c5-5j5r5" event={"ID":"36f8f834-bec7-411d-a4bc-86c141eecc14","Type":"ContainerDied","Data":"4b872e6c0c74a39c0cdc704903b77270ca773f2c578fad8bf89a95457c71232f"}
Oct 30 22:36:42.709272 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:42.708972    4477 scope.go:122] "RemoveContainer" containerID="0a591696b32e0c890267bf149eb77fc8887c4444f0c9dde3ec63f522bc8c5296"
Oct 30 22:36:42.724080 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:42.723796    4477 scope.go:122] "RemoveContainer" containerID="0a591696b32e0c890267bf149eb77fc8887c4444f0c9dde3ec63f522bc8c5296"
Oct 30 22:36:42.724188 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:36:42.724152    4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a591696b32e0c890267bf149eb77fc8887c4444f0c9dde3ec63f522bc8c5296\": container with ID starting with 0a591696b32e0c890267bf149eb77fc8887c4444f0c9dde3ec63f522bc8c5296 not found: ID does not exist" containerID="0a591696b32e0c890267bf149eb77fc8887c4444f0c9dde3ec63f522bc8c5296"
Oct 30 22:36:42.724188 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:42.724179    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a591696b32e0c890267bf149eb77fc8887c4444f0c9dde3ec63f522bc8c5296"} err="failed to get container status \"0a591696b32e0c890267bf149eb77fc8887c4444f0c9dde3ec63f522bc8c5296\": rpc error: code = NotFound desc = could not find container \"0a591696b32e0c890267bf149eb77fc8887c4444f0c9dde3ec63f522bc8c5296\": container with ID starting with 0a591696b32e0c890267bf149eb77fc8887c4444f0c9dde3ec63f522bc8c5296 not found: ID does not exist"
Oct 30 22:36:42.728881 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:42.727132    4477 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36f8f834-bec7-411d-a4bc-86c141eecc14-metrics-certs\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:36:42.728881 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:42.727158    4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nw72p\" (UniqueName: \"kubernetes.io/projected/36f8f834-bec7-411d-a4bc-86c141eecc14-kube-api-access-nw72p\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:36:42.728881 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:42.727172    4477 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36f8f834-bec7-411d-a4bc-86c141eecc14-maas-api-tls\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:36:42.728881 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:42.727185    4477 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36f8f834-bec7-411d-a4bc-86c141eecc14-openshift-service-ca\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:36:42.768633 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:42.768591    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4f7439-85d59f65c5-5j5r5"]
Oct 30 22:36:42.778018 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:42.777983    4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4f7439-85d59f65c5-5j5r5"]
Oct 30 22:36:44.194699 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:44.194664    4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36f8f834-bec7-411d-a4bc-86c141eecc14" path="/var/lib/kubelet/pods/36f8f834-bec7-411d-a4bc-86c141eecc14/volumes"
Oct 30 22:36:45.171564 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:45.171533    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-x68sw"]
Oct 30 22:36:45.172354 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:45.172332    4477 cpu_manager.go:414] "containerMap: removing container" podUID="36f8f834-bec7-411d-a4bc-86c141eecc14" containerName="maas-api"
Oct 30 22:36:45.172354 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:45.172351    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36f8f834-bec7-411d-a4bc-86c141eecc14" containerName="maas-api"
Oct 30 22:36:45.172522 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:45.172451    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="36f8f834-bec7-411d-a4bc-86c141eecc14" containerName="maas-api"
Oct 30 22:36:45.177491 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:45.177471    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-x68sw"
Oct 30 22:36:45.212712 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:45.212679    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-x68sw"]
Oct 30 22:36:45.256373 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:45.256346    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca4b71ad-f474-4db6-a350-8f1da8fe6244-istio-token\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-x68sw\" (UID: \"ca4b71ad-f474-4db6-a350-8f1da8fe6244\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-x68sw"
Oct 30 22:36:45.256495 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:45.256404    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca4b71ad-f474-4db6-a350-8f1da8fe6244-credential-socket\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-x68sw\" (UID: \"ca4b71ad-f474-4db6-a350-8f1da8fe6244\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-x68sw"
Oct 30 22:36:45.256562 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:45.256545    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca4b71ad-f474-4db6-a350-8f1da8fe6244-workload-certs\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-x68sw\" (UID: \"ca4b71ad-f474-4db6-a350-8f1da8fe6244\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-x68sw"
Oct 30 22:36:45.256620 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:45.256602    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca4b71ad-f474-4db6-a350-8f1da8fe6244-istiod-ca-cert\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-x68sw\" (UID: \"ca4b71ad-f474-4db6-a350-8f1da8fe6244\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-x68sw"
Oct 30 22:36:45.256673 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:45.256660    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca4b71ad-f474-4db6-a350-8f1da8fe6244-istio-data\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-x68sw\" (UID: \"ca4b71ad-f474-4db6-a350-8f1da8fe6244\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-x68sw"
Oct 30 22:36:45.256707 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:45.256701    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca4b71ad-f474-4db6-a350-8f1da8fe6244-istio-envoy\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-x68sw\" (UID: \"ca4b71ad-f474-4db6-a350-8f1da8fe6244\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-x68sw"
Oct 30 22:36:45.256741 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:45.256716    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6jlq\" (UniqueName: \"kubernetes.io/projected/ca4b71ad-f474-4db6-a350-8f1da8fe6244-kube-api-access-n6jlq\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-x68sw\" (UID: \"ca4b71ad-f474-4db6-a350-8f1da8fe6244\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-x68sw"
Oct 30 22:36:45.256829 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:45.256815    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca4b71ad-f474-4db6-a350-8f1da8fe6244-workload-socket\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-x68sw\" (UID: \"ca4b71ad-f474-4db6-a350-8f1da8fe6244\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-x68sw"
Oct 30 22:36:45.256860 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:45.256847    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca4b71ad-f474-4db6-a350-8f1da8fe6244-istio-podinfo\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-x68sw\" (UID: \"ca4b71ad-f474-4db6-a350-8f1da8fe6244\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-x68sw"
Oct 30 22:36:45.358622 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:45.358590    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca4b71ad-f474-4db6-a350-8f1da8fe6244-credential-socket\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-x68sw\" (UID: \"ca4b71ad-f474-4db6-a350-8f1da8fe6244\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-x68sw"
Oct 30 22:36:45.359828 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:45.359803    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca4b71ad-f474-4db6-a350-8f1da8fe6244-workload-certs\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-x68sw\" (UID: \"ca4b71ad-f474-4db6-a350-8f1da8fe6244\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-x68sw"
Oct 30 22:36:45.360750 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:45.360725    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca4b71ad-f474-4db6-a350-8f1da8fe6244-workload-certs\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-x68sw\" (UID: \"ca4b71ad-f474-4db6-a350-8f1da8fe6244\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-x68sw"
Oct 30 22:36:45.360871 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:45.360735    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca4b71ad-f474-4db6-a350-8f1da8fe6244-istiod-ca-cert\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-x68sw\" (UID: \"ca4b71ad-f474-4db6-a350-8f1da8fe6244\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-x68sw"
Oct 30 22:36:45.361007 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:45.360989    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca4b71ad-f474-4db6-a350-8f1da8fe6244-istio-data\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-x68sw\" (UID: \"ca4b71ad-f474-4db6-a350-8f1da8fe6244\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-x68sw"
Oct 30 22:36:45.361127 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:45.360981    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca4b71ad-f474-4db6-a350-8f1da8fe6244-credential-socket\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-x68sw\" (UID: \"ca4b71ad-f474-4db6-a350-8f1da8fe6244\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-x68sw"
Oct 30 22:36:45.361298 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:45.361191    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca4b71ad-f474-4db6-a350-8f1da8fe6244-istio-envoy\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-x68sw\" (UID: \"ca4b71ad-f474-4db6-a350-8f1da8fe6244\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-x68sw"
Oct 30 22:36:45.361298 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:45.361269    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca4b71ad-f474-4db6-a350-8f1da8fe6244-istio-data\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-x68sw\" (UID: \"ca4b71ad-f474-4db6-a350-8f1da8fe6244\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-x68sw"
Oct 30 22:36:45.361453 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:45.361347    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6jlq\" (UniqueName: \"kubernetes.io/projected/ca4b71ad-f474-4db6-a350-8f1da8fe6244-kube-api-access-n6jlq\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-x68sw\" (UID: \"ca4b71ad-f474-4db6-a350-8f1da8fe6244\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-x68sw"
Oct 30 22:36:45.361534 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:45.361448    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca4b71ad-f474-4db6-a350-8f1da8fe6244-workload-socket\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-x68sw\" (UID: \"ca4b71ad-f474-4db6-a350-8f1da8fe6244\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-x68sw"
Oct 30 22:36:45.362295 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:45.361476    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca4b71ad-f474-4db6-a350-8f1da8fe6244-istio-podinfo\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-x68sw\" (UID: \"ca4b71ad-f474-4db6-a350-8f1da8fe6244\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-x68sw"
Oct 30 22:36:45.362606 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:45.362582    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca4b71ad-f474-4db6-a350-8f1da8fe6244-istio-token\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-x68sw\" (UID: \"ca4b71ad-f474-4db6-a350-8f1da8fe6244\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-x68sw"
Oct 30 22:36:45.362837 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:45.362814    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca4b71ad-f474-4db6-a350-8f1da8fe6244-workload-socket\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-x68sw\" (UID: \"ca4b71ad-f474-4db6-a350-8f1da8fe6244\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-x68sw"
Oct 30 22:36:45.362994 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:45.361589    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca4b71ad-f474-4db6-a350-8f1da8fe6244-istiod-ca-cert\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-x68sw\" (UID: \"ca4b71ad-f474-4db6-a350-8f1da8fe6244\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-x68sw"
Oct 30 22:36:45.365244 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:45.365220    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca4b71ad-f474-4db6-a350-8f1da8fe6244-istio-podinfo\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-x68sw\" (UID: \"ca4b71ad-f474-4db6-a350-8f1da8fe6244\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-x68sw"
Oct 30 22:36:45.366647 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:45.366600    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca4b71ad-f474-4db6-a350-8f1da8fe6244-istio-envoy\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-x68sw\" (UID: \"ca4b71ad-f474-4db6-a350-8f1da8fe6244\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-x68sw"
Oct 30 22:36:45.391609 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:45.391545    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6jlq\" (UniqueName: \"kubernetes.io/projected/ca4b71ad-f474-4db6-a350-8f1da8fe6244-kube-api-access-n6jlq\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-x68sw\" (UID: \"ca4b71ad-f474-4db6-a350-8f1da8fe6244\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-x68sw"
Oct 30 22:36:45.392954 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:45.392912    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca4b71ad-f474-4db6-a350-8f1da8fe6244-istio-token\") pod \"e2e-inf-a-6922d3-openshift-default-689b5f95bc-x68sw\" (UID: \"ca4b71ad-f474-4db6-a350-8f1da8fe6244\") " pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-x68sw"
Oct 30 22:36:45.491744 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:45.491652    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-x68sw"
Oct 30 22:36:45.817325 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:45.817295    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-x68sw"]
Oct 30 22:36:45.819569 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:36:45.819540    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca4b71ad_f474_4db6_a350_8f1da8fe6244.slice/crio-ba8d08b4641c31efb55b3a07f60e1cfaa8a7def418dcd98b20aab78f1507a7db WatchSource:0}: Error finding container ba8d08b4641c31efb55b3a07f60e1cfaa8a7def418dcd98b20aab78f1507a7db: Status 404 returned error can't find the container with id ba8d08b4641c31efb55b3a07f60e1cfaa8a7def418dcd98b20aab78f1507a7db
Oct 30 22:36:45.822609 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:45.822569    4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:36:45.822724 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:45.822653    4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:36:45.822724 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:45.822697    4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:36:46.735630 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:46.735593    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-x68sw" event={"ID":"ca4b71ad-f474-4db6-a350-8f1da8fe6244","Type":"ContainerStarted","Data":"64627c9156678f089e05be2217551c32c7a9e7303a7f30a797dc1d24c08235b5"}
Oct 30 22:36:46.735999 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:46.735648    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-x68sw" event={"ID":"ca4b71ad-f474-4db6-a350-8f1da8fe6244","Type":"ContainerStarted","Data":"ba8d08b4641c31efb55b3a07f60e1cfaa8a7def418dcd98b20aab78f1507a7db"}
Oct 30 22:36:46.805563 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:46.805514    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-x68sw" podStartSLOduration=1.805500702 podStartE2EDuration="1.805500702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:36:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:36:46.792140785 +0000 UTC m=+1790.718066991" watchObservedRunningTime="2026-10-30 22:36:46.805500702 +0000 UTC m=+1790.731426907"
Oct 30 22:36:47.494263 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:47.494220    4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-x68sw"
Oct 30 22:36:47.496342 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:47.496309    4477 patch_prober.go:29] interesting pod/e2e-inf-a-6922d3-openshift-default-689b5f95bc-x68sw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body=
Oct 30 22:36:47.496514 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:47.496370    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-x68sw" podUID="ca4b71ad-f474-4db6-a350-8f1da8fe6244" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused"
Oct 30 22:36:48.493382 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:48.493343    4477 patch_prober.go:29] interesting pod/e2e-inf-a-6922d3-openshift-default-689b5f95bc-x68sw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body=
Oct 30 22:36:48.493953 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:48.493921    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-x68sw" podUID="ca4b71ad-f474-4db6-a350-8f1da8fe6244" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused"
Oct 30 22:36:49.494025 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:49.493991    4477 patch_prober.go:29] interesting pod/e2e-inf-a-6922d3-openshift-default-689b5f95bc-x68sw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body=
Oct 30 22:36:49.494557 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:49.494199    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-x68sw" podUID="ca4b71ad-f474-4db6-a350-8f1da8fe6244" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused"
Oct 30 22:36:50.494453 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:50.494421    4477 patch_prober.go:29] interesting pod/e2e-inf-a-6922d3-openshift-default-689b5f95bc-x68sw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body=
Oct 30 22:36:50.494867 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:50.494475    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-x68sw" podUID="ca4b71ad-f474-4db6-a350-8f1da8fe6244" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused"
Oct 30 22:36:51.492011 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:51.491975    4477 patch_prober.go:29] interesting pod/e2e-inf-a-6922d3-openshift-default-689b5f95bc-x68sw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body=
Oct 30 22:36:51.492333 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:51.492294    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-x68sw" podUID="ca4b71ad-f474-4db6-a350-8f1da8fe6244" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused"
Oct 30 22:36:52.493354 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:52.493309    4477 patch_prober.go:29] interesting pod/e2e-inf-a-6922d3-openshift-default-689b5f95bc-x68sw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body=
Oct 30 22:36:52.494031 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:52.493972    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-x68sw" podUID="ca4b71ad-f474-4db6-a350-8f1da8fe6244" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused"
Oct 30 22:36:53.496835 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:53.496803    4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-x68sw"
Oct 30 22:36:53.497333 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:53.497299    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-x68sw"
Oct 30 22:36:53.498018 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:53.497997    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-x68sw"
Oct 30 22:36:56.280392 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:36:56.280355    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 22:36:56.861488 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:56.861461    4477 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jcf8l_015f218d-bf1f-4e7e-bb5d-b805af003246/ovn-acl-logging/0.log"
Oct 30 22:36:56.909566 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:56.909541    4477 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jcf8l_015f218d-bf1f-4e7e-bb5d-b805af003246/ovn-acl-logging/0.log"
Oct 30 22:36:59.405498 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:59.405463    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-62236a-7fc8b485b4-tspl7"]
Oct 30 22:36:59.405973 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:59.405693    4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-62236a-7fc8b485b4-tspl7" podUID="d5aad55c-1b94-460e-96e3-a40d87b421e0" containerName="maas-api" containerID="cri-o://b6b7fb24d8968f0aea9b19993483235d19e958e5bd3e0239bde95c454aeb8a55" gracePeriod=30
Oct 30 22:36:59.769276 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:59.769241    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-62236a-7fc8b485b4-tspl7"
Oct 30 22:36:59.862114 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:59.862078    4477 generic.go:356] "Generic (PLEG): container finished" podID="d5aad55c-1b94-460e-96e3-a40d87b421e0" containerID="b6b7fb24d8968f0aea9b19993483235d19e958e5bd3e0239bde95c454aeb8a55" exitCode=0
Oct 30 22:36:59.862114 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:59.862113    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-62236a-7fc8b485b4-tspl7"
Oct 30 22:36:59.862337 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:59.862137    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-62236a-7fc8b485b4-tspl7" event={"ID":"d5aad55c-1b94-460e-96e3-a40d87b421e0","Type":"ContainerDied","Data":"b6b7fb24d8968f0aea9b19993483235d19e958e5bd3e0239bde95c454aeb8a55"}
Oct 30 22:36:59.862337 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:59.862204    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-62236a-7fc8b485b4-tspl7" event={"ID":"d5aad55c-1b94-460e-96e3-a40d87b421e0","Type":"ContainerDied","Data":"4e71befa99319e97f7f873bd4b01061254d99fd5999498aa5e06c58f4d1083f5"}
Oct 30 22:36:59.862337 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:59.862233    4477 scope.go:122] "RemoveContainer" containerID="b6b7fb24d8968f0aea9b19993483235d19e958e5bd3e0239bde95c454aeb8a55"
Oct 30 22:36:59.878360 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:59.878339    4477 scope.go:122] "RemoveContainer" containerID="b6b7fb24d8968f0aea9b19993483235d19e958e5bd3e0239bde95c454aeb8a55"
Oct 30 22:36:59.878684 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:36:59.878663    4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6b7fb24d8968f0aea9b19993483235d19e958e5bd3e0239bde95c454aeb8a55\": container with ID starting with b6b7fb24d8968f0aea9b19993483235d19e958e5bd3e0239bde95c454aeb8a55 not found: ID does not exist" containerID="b6b7fb24d8968f0aea9b19993483235d19e958e5bd3e0239bde95c454aeb8a55"
Oct 30 22:36:59.878776 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:59.878696    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6b7fb24d8968f0aea9b19993483235d19e958e5bd3e0239bde95c454aeb8a55"} err="failed to get container status \"b6b7fb24d8968f0aea9b19993483235d19e958e5bd3e0239bde95c454aeb8a55\": rpc error: code = NotFound desc = could not find container \"b6b7fb24d8968f0aea9b19993483235d19e958e5bd3e0239bde95c454aeb8a55\": container with ID starting with b6b7fb24d8968f0aea9b19993483235d19e958e5bd3e0239bde95c454aeb8a55 not found: ID does not exist"
Oct 30 22:36:59.895335 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:59.895307    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5aad55c-1b94-460e-96e3-a40d87b421e0-kube-api-access-zxdwk\" (UniqueName: \"kubernetes.io/projected/d5aad55c-1b94-460e-96e3-a40d87b421e0-kube-api-access-zxdwk\") pod \"d5aad55c-1b94-460e-96e3-a40d87b421e0\" (UID: \"d5aad55c-1b94-460e-96e3-a40d87b421e0\") "
Oct 30 22:36:59.895458 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:59.895374    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5aad55c-1b94-460e-96e3-a40d87b421e0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5aad55c-1b94-460e-96e3-a40d87b421e0-metrics-certs\") pod \"d5aad55c-1b94-460e-96e3-a40d87b421e0\" (UID: \"d5aad55c-1b94-460e-96e3-a40d87b421e0\") "
Oct 30 22:36:59.895458 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:59.895453    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5aad55c-1b94-460e-96e3-a40d87b421e0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5aad55c-1b94-460e-96e3-a40d87b421e0-maas-api-tls\") pod \"d5aad55c-1b94-460e-96e3-a40d87b421e0\" (UID: \"d5aad55c-1b94-460e-96e3-a40d87b421e0\") "
Oct 30 22:36:59.895667 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:59.895644    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5aad55c-1b94-460e-96e3-a40d87b421e0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5aad55c-1b94-460e-96e3-a40d87b421e0-openshift-service-ca\") pod \"d5aad55c-1b94-460e-96e3-a40d87b421e0\" (UID: \"d5aad55c-1b94-460e-96e3-a40d87b421e0\") "
Oct 30 22:36:59.896382 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:59.896335    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5aad55c-1b94-460e-96e3-a40d87b421e0-openshift-service-ca" pod "d5aad55c-1b94-460e-96e3-a40d87b421e0" (UID: "d5aad55c-1b94-460e-96e3-a40d87b421e0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:36:59.897947 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:59.897921    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5aad55c-1b94-460e-96e3-a40d87b421e0-maas-api-tls" pod "d5aad55c-1b94-460e-96e3-a40d87b421e0" (UID: "d5aad55c-1b94-460e-96e3-a40d87b421e0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:36:59.898444 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:59.898424    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5aad55c-1b94-460e-96e3-a40d87b421e0-kube-api-access-zxdwk" pod "d5aad55c-1b94-460e-96e3-a40d87b421e0" (UID: "d5aad55c-1b94-460e-96e3-a40d87b421e0"). InnerVolumeSpecName "kube-api-access-zxdwk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:36:59.899442 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:59.899418    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5aad55c-1b94-460e-96e3-a40d87b421e0-metrics-certs" pod "d5aad55c-1b94-460e-96e3-a40d87b421e0" (UID: "d5aad55c-1b94-460e-96e3-a40d87b421e0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:36:59.998508 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:59.998468    4477 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5aad55c-1b94-460e-96e3-a40d87b421e0-maas-api-tls\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:36:59.998508 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:59.998509    4477 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5aad55c-1b94-460e-96e3-a40d87b421e0-openshift-service-ca\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:36:59.998750 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:59.998524    4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxdwk\" (UniqueName: \"kubernetes.io/projected/d5aad55c-1b94-460e-96e3-a40d87b421e0-kube-api-access-zxdwk\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:36:59.998750 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:36:59.998539    4477 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5aad55c-1b94-460e-96e3-a40d87b421e0-metrics-certs\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:37:00.337066 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:37:00.337021    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-62236a-7fc8b485b4-tspl7"]
Oct 30 22:37:00.348254 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:37:00.348222    4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-62236a-7fc8b485b4-tspl7"]
Oct 30 22:37:00.457494 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:37:00.457457    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-40229e-f6c86585d-gdhkl"]
Oct 30 22:37:00.458591 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:37:00.458540    4477 cpu_manager.go:414] "containerMap: removing container" podUID="d5aad55c-1b94-460e-96e3-a40d87b421e0" containerName="maas-api"
Oct 30 22:37:00.458591 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:37:00.458563    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5aad55c-1b94-460e-96e3-a40d87b421e0" containerName="maas-api"
Oct 30 22:37:00.458817 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:37:00.458799    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5aad55c-1b94-460e-96e3-a40d87b421e0" containerName="maas-api"
Oct 30 22:37:00.464366 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:37:00.464347    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-40229e-f6c86585d-gdhkl"
Oct 30 22:37:00.478623 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:37:00.478588    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-40229e-serving-cert\""
Oct 30 22:37:00.482215 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:37:00.482190    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-40229e-f6c86585d-gdhkl"]
Oct 30 22:37:00.523518 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:37:00.523488    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c927c80-1da0-4b5b-8655-c67e8a35f000-openshift-service-ca\") pod \"maas-api-e2e-inf-a-40229e-f6c86585d-gdhkl\" (UID: \"8c927c80-1da0-4b5b-8655-c67e8a35f000\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-40229e-f6c86585d-gdhkl"
Oct 30 22:37:00.523676 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:37:00.523644    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c927c80-1da0-4b5b-8655-c67e8a35f000-maas-api-tls\") pod \"maas-api-e2e-inf-a-40229e-f6c86585d-gdhkl\" (UID: \"8c927c80-1da0-4b5b-8655-c67e8a35f000\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-40229e-f6c86585d-gdhkl"
Oct 30 22:37:00.523733 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:37:00.523708    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbkrd\" (UniqueName: \"kubernetes.io/projected/8c927c80-1da0-4b5b-8655-c67e8a35f000-kube-api-access-mbkrd\") pod \"maas-api-e2e-inf-a-40229e-f6c86585d-gdhkl\" (UID: \"8c927c80-1da0-4b5b-8655-c67e8a35f000\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-40229e-f6c86585d-gdhkl"
Oct 30 22:37:00.523860 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:37:00.523833    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c927c80-1da0-4b5b-8655-c67e8a35f000-metrics-certs\") pod \"maas-api-e2e-inf-a-40229e-f6c86585d-gdhkl\" (UID: \"8c927c80-1da0-4b5b-8655-c67e8a35f000\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-40229e-f6c86585d-gdhkl"
Oct 30 22:37:00.624968 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:37:00.624941    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c927c80-1da0-4b5b-8655-c67e8a35f000-openshift-service-ca\") pod \"maas-api-e2e-inf-a-40229e-f6c86585d-gdhkl\" (UID: \"8c927c80-1da0-4b5b-8655-c67e8a35f000\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-40229e-f6c86585d-gdhkl"
Oct 30 22:37:00.625159 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:37:00.625019    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c927c80-1da0-4b5b-8655-c67e8a35f000-maas-api-tls\") pod \"maas-api-e2e-inf-a-40229e-f6c86585d-gdhkl\" (UID: \"8c927c80-1da0-4b5b-8655-c67e8a35f000\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-40229e-f6c86585d-gdhkl"
Oct 30 22:37:00.625159 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:37:00.625070    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbkrd\" (UniqueName: \"kubernetes.io/projected/8c927c80-1da0-4b5b-8655-c67e8a35f000-kube-api-access-mbkrd\") pod \"maas-api-e2e-inf-a-40229e-f6c86585d-gdhkl\" (UID: \"8c927c80-1da0-4b5b-8655-c67e8a35f000\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-40229e-f6c86585d-gdhkl"
Oct 30 22:37:00.625159 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:37:00.625140    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c927c80-1da0-4b5b-8655-c67e8a35f000-metrics-certs\") pod \"maas-api-e2e-inf-a-40229e-f6c86585d-gdhkl\" (UID: \"8c927c80-1da0-4b5b-8655-c67e8a35f000\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-40229e-f6c86585d-gdhkl"
Oct 30 22:37:00.625322 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:37:00.625190    4477 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-40229e-serving-cert: secret "maas-api-e2e-inf-a-40229e-serving-cert" not found
Oct 30 22:37:00.625322 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:37:00.625261    4477 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c927c80-1da0-4b5b-8655-c67e8a35f000-maas-api-tls podName:8c927c80-1da0-4b5b-8655-c67e8a35f000 nodeName:}" failed. No retries permitted until 2026-10-30 22:37:01.125242178 +0000 UTC m=+1805.051168381 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8c927c80-1da0-4b5b-8655-c67e8a35f000-maas-api-tls") pod "maas-api-e2e-inf-a-40229e-f6c86585d-gdhkl" (UID: "8c927c80-1da0-4b5b-8655-c67e8a35f000") : secret "maas-api-e2e-inf-a-40229e-serving-cert" not found
Oct 30 22:37:00.625806 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:37:00.625777    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c927c80-1da0-4b5b-8655-c67e8a35f000-openshift-service-ca\") pod \"maas-api-e2e-inf-a-40229e-f6c86585d-gdhkl\" (UID: \"8c927c80-1da0-4b5b-8655-c67e8a35f000\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-40229e-f6c86585d-gdhkl"
Oct 30 22:37:00.627747 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:37:00.627722    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c927c80-1da0-4b5b-8655-c67e8a35f000-metrics-certs\") pod \"maas-api-e2e-inf-a-40229e-f6c86585d-gdhkl\" (UID: \"8c927c80-1da0-4b5b-8655-c67e8a35f000\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-40229e-f6c86585d-gdhkl"
Oct 30 22:37:00.648072 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:37:00.644936    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbkrd\" (UniqueName: \"kubernetes.io/projected/8c927c80-1da0-4b5b-8655-c67e8a35f000-kube-api-access-mbkrd\") pod \"maas-api-e2e-inf-a-40229e-f6c86585d-gdhkl\" (UID: \"8c927c80-1da0-4b5b-8655-c67e8a35f000\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-40229e-f6c86585d-gdhkl"
Oct 30 22:37:01.130905 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:37:01.130876    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c927c80-1da0-4b5b-8655-c67e8a35f000-maas-api-tls\") pod \"maas-api-e2e-inf-a-40229e-f6c86585d-gdhkl\" (UID: \"8c927c80-1da0-4b5b-8655-c67e8a35f000\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-40229e-f6c86585d-gdhkl"
Oct 30 22:37:01.133140 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:37:01.133118    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c927c80-1da0-4b5b-8655-c67e8a35f000-maas-api-tls\") pod \"maas-api-e2e-inf-a-40229e-f6c86585d-gdhkl\" (UID: \"8c927c80-1da0-4b5b-8655-c67e8a35f000\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-40229e-f6c86585d-gdhkl"
Oct 30 22:37:01.316094 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:37:01.313382    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-bdd94b/test-model-bdd94b-kserve-86fcb699c-svmrn"]
Oct 30 22:37:01.318417 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:37:01.318393    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-bdd94b/test-model-bdd94b-kserve-86fcb699c-svmrn"
Oct 30 22:37:01.337978 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:37:01.335449    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-bdd94b\"/\"default-dockercfg-dlxwn\""
Oct 30 22:37:01.337978 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:37:01.336109    4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-bdd94b\"/\"kube-root-ca.crt\""
Oct 30 22:37:01.337978 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:37:01.336180    4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-bdd94b\"/\"openshift-service-ca.crt\""
Oct 30 22:37:01.341276 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:37:01.340877    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-bdd94b/test-model-bdd94b-kserve-86fcb699c-svmrn"]
Oct 30 22:37:01.346177 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:37:01.346152    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-bdd94b\"/\"test-model-bdd94b-kserve-self-signed-certs\""
Oct 30 22:37:01.381345 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:37:01.381285    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-40229e-f6c86585d-gdhkl"
Oct 30 22:37:01.438724 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:37:01.438687    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a0bd85e-66f4-4942-840a-7f91e0413a8a-model-cache\") pod \"test-model-bdd94b-kserve-86fcb699c-svmrn\" (UID: \"6a0bd85e-66f4-4942-840a-7f91e0413a8a\") " pod="ai-tenant-e2e-inf-a-bdd94b/test-model-bdd94b-kserve-86fcb699c-svmrn"
Oct 30 22:37:01.438891 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:37:01.438744    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a0bd85e-66f4-4942-840a-7f91e0413a8a-tls-certs\") pod \"test-model-bdd94b-kserve-86fcb699c-svmrn\" (UID: \"6a0bd85e-66f4-4942-840a-7f91e0413a8a\") " pod="ai-tenant-e2e-inf-a-bdd94b/test-model-bdd94b-kserve-86fcb699c-svmrn"
Oct 30 22:37:01.438950 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:37:01.438901    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a0bd85e-66f4-4942-840a-7f91e0413a8a-dshm\") pod \"test-model-bdd94b-kserve-86fcb699c-svmrn\" (UID: \"6a0bd85e-66f4-4942-840a-7f91e0413a8a\") " pod="ai-tenant-e2e-inf-a-bdd94b/test-model-bdd94b-kserve-86fcb699c-svmrn"
Oct 30 22:37:01.439014 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:37:01.438955    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9fwm\" (UniqueName: \"kubernetes.io/projected/6a0bd85e-66f4-4942-840a-7f91e0413a8a-kube-api-access-l9fwm\") pod \"test-model-bdd94b-kserve-86fcb699c-svmrn\" (UID: \"6a0bd85e-66f4-4942-840a-7f91e0413a8a\") " pod="ai-tenant-e2e-inf-a-bdd94b/test-model-bdd94b-kserve-86fcb699c-svmrn"
Oct 30 22:37:01.439206 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:37:01.439182    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a0bd85e-66f4-4942-840a-7f91e0413a8a-home\") pod \"test-model-bdd94b-kserve-86fcb699c-svmrn\" (UID: \"6a0bd85e-66f4-4942-840a-7f91e0413a8a\") " pod="ai-tenant-e2e-inf-a-bdd94b/test-model-bdd94b-kserve-86fcb699c-svmrn"
Oct 30 22:37:01.439271 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:37:01.439244    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a0bd85e-66f4-4942-840a-7f91e0413a8a-tmp-dir\") pod \"test-model-bdd94b-kserve-86fcb699c-svmrn\" (UID: \"6a0bd85e-66f4-4942-840a-7f91e0413a8a\") " pod="ai-tenant-e2e-inf-a-bdd94b/test-model-bdd94b-kserve-86fcb699c-svmrn"
Oct 30 22:37:01.541004 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:37:01.540952    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a0bd85e-66f4-4942-840a-7f91e0413a8a-dshm\") pod \"test-model-bdd94b-kserve-86fcb699c-svmrn\" (UID: \"6a0bd85e-66f4-4942-840a-7f91e0413a8a\") " pod="ai-tenant-e2e-inf-a-bdd94b/test-model-bdd94b-kserve-86fcb699c-svmrn"
Oct 30 22:37:01.541504 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:37:01.541012    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9fwm\" (UniqueName: \"kubernetes.io/projected/6a0bd85e-66f4-4942-840a-7f91e0413a8a-kube-api-access-l9fwm\") pod \"test-model-bdd94b-kserve-86fcb699c-svmrn\" (UID: \"6a0bd85e-66f4-4942-840a-7f91e0413a8a\") " pod="ai-tenant-e2e-inf-a-bdd94b/test-model-bdd94b-kserve-86fcb699c-svmrn"
Oct 30 22:37:01.541504 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:37:01.541176    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a0bd85e-66f4-4942-840a-7f91e0413a8a-home\") pod \"test-model-bdd94b-kserve-86fcb699c-svmrn\" (UID: \"6a0bd85e-66f4-4942-840a-7f91e0413a8a\") " pod="ai-tenant-e2e-inf-a-bdd94b/test-model-bdd94b-kserve-86fcb699c-svmrn"
Oct 30 22:37:01.541504 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:37:01.541234    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a0bd85e-66f4-4942-840a-7f91e0413a8a-tmp-dir\") pod \"test-model-bdd94b-kserve-86fcb699c-svmrn\" (UID: \"6a0bd85e-66f4-4942-840a-7f91e0413a8a\") " pod="ai-tenant-e2e-inf-a-bdd94b/test-model-bdd94b-kserve-86fcb699c-svmrn"
Oct 30 22:37:01.541504 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:37:01.541431    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a0bd85e-66f4-4942-840a-7f91e0413a8a-model-cache\") pod \"test-model-bdd94b-kserve-86fcb699c-svmrn\" (UID: \"6a0bd85e-66f4-4942-840a-7f91e0413a8a\") " pod="ai-tenant-e2e-inf-a-bdd94b/test-model-bdd94b-kserve-86fcb699c-svmrn"
Oct 30 22:37:01.541504 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:37:01.541469    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a0bd85e-66f4-4942-840a-7f91e0413a8a-tls-certs\") pod \"test-model-bdd94b-kserve-86fcb699c-svmrn\" (UID: \"6a0bd85e-66f4-4942-840a-7f91e0413a8a\") " pod="ai-tenant-e2e-inf-a-bdd94b/test-model-bdd94b-kserve-86fcb699c-svmrn"
Oct 30 22:37:01.541829 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:37:01.541793    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a0bd85e-66f4-4942-840a-7f91e0413a8a-home\") pod \"test-model-bdd94b-kserve-86fcb699c-svmrn\" (UID: \"6a0bd85e-66f4-4942-840a-7f91e0413a8a\") " pod="ai-tenant-e2e-inf-a-bdd94b/test-model-bdd94b-kserve-86fcb699c-svmrn"
Oct 30 22:37:01.542386 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:37:01.542328    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a0bd85e-66f4-4942-840a-7f91e0413a8a-tmp-dir\") pod \"test-model-bdd94b-kserve-86fcb699c-svmrn\" (UID: \"6a0bd85e-66f4-4942-840a-7f91e0413a8a\") " pod="ai-tenant-e2e-inf-a-bdd94b/test-model-bdd94b-kserve-86fcb699c-svmrn"
Oct 30 22:37:01.542521 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:37:01.542492    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a0bd85e-66f4-4942-840a-7f91e0413a8a-model-cache\") pod \"test-model-bdd94b-kserve-86fcb699c-svmrn\" (UID: \"6a0bd85e-66f4-4942-840a-7f91e0413a8a\") " pod="ai-tenant-e2e-inf-a-bdd94b/test-model-bdd94b-kserve-86fcb699c-svmrn"
Oct 30 22:37:01.544771 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:37:01.544744    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a0bd85e-66f4-4942-840a-7f91e0413a8a-tls-certs\") pod \"test-model-bdd94b-kserve-86fcb699c-svmrn\" (UID: \"6a0bd85e-66f4-4942-840a-7f91e0413a8a\") " pod="ai-tenant-e2e-inf-a-bdd94b/test-model-bdd94b-kserve-86fcb699c-svmrn"
Oct 30 22:37:01.544912 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:37:01.544772    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a0bd85e-66f4-4942-840a-7f91e0413a8a-dshm\") pod \"test-model-bdd94b-kserve-86fcb699c-svmrn\" (UID: \"6a0bd85e-66f4-4942-840a-7f91e0413a8a\") " pod="ai-tenant-e2e-inf-a-bdd94b/test-model-bdd94b-kserve-86fcb699c-svmrn"
Oct 30 22:37:01.563320 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:37:01.563287    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9fwm\" (UniqueName: \"kubernetes.io/projected/6a0bd85e-66f4-4942-840a-7f91e0413a8a-kube-api-access-l9fwm\") pod \"test-model-bdd94b-kserve-86fcb699c-svmrn\" (UID: \"6a0bd85e-66f4-4942-840a-7f91e0413a8a\") " pod="ai-tenant-e2e-inf-a-bdd94b/test-model-bdd94b-kserve-86fcb699c-svmrn"
Oct 30 22:37:01.649109 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:37:01.647612    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-bdd94b/test-model-bdd94b-kserve-86fcb699c-svmrn"
Oct 30 22:37:01.722072 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:37:01.721009    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-40229e-f6c86585d-gdhkl"]
Oct 30 22:37:01.728069 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:37:01.726932    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c927c80_1da0_4b5b_8655_c67e8a35f000.slice/crio-741f2711cf8d427ef6e6d33d9d379de35473b55ef6649938809619ba9d3ac052 WatchSource:0}: Error finding container 741f2711cf8d427ef6e6d33d9d379de35473b55ef6649938809619ba9d3ac052: Status 404 returned error can't find the container with id 741f2711cf8d427ef6e6d33d9d379de35473b55ef6649938809619ba9d3ac052
Oct 30 22:37:01.996142 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:37:01.995544    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-40229e-f6c86585d-gdhkl" event={"ID":"8c927c80-1da0-4b5b-8655-c67e8a35f000","Type":"ContainerStarted","Data":"741f2711cf8d427ef6e6d33d9d379de35473b55ef6649938809619ba9d3ac052"}
Oct 30 22:37:02.138028 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:37:02.138000    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-bdd94b/test-model-bdd94b-kserve-86fcb699c-svmrn"]
Oct 30 22:37:02.139334 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:37:02.139303    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6a0bd85e_66f4_4942_840a_7f91e0413a8a.slice/crio-e2b5e75f0c2d37d7227d4b1807225aebb2f464443b4b2dbeeee45faa01d29ec5 WatchSource:0}: Error finding container e2b5e75f0c2d37d7227d4b1807225aebb2f464443b4b2dbeeee45faa01d29ec5: Status 404 returned error can't find the container with id e2b5e75f0c2d37d7227d4b1807225aebb2f464443b4b2dbeeee45faa01d29ec5
Oct 30 22:37:02.196929 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:37:02.196471    4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5aad55c-1b94-460e-96e3-a40d87b421e0" path="/var/lib/kubelet/pods/d5aad55c-1b94-460e-96e3-a40d87b421e0/volumes"
Oct 30 22:37:03.008949 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:37:03.008912    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-bdd94b/test-model-bdd94b-kserve-86fcb699c-svmrn" event={"ID":"6a0bd85e-66f4-4942-840a-7f91e0413a8a","Type":"ContainerStarted","Data":"7bbd121e65b9d9201b3ff4d5acc00ec0160a6fd877efd16006e871192b0c9966"}
Oct 30 22:37:03.009466 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:37:03.008966    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-bdd94b/test-model-bdd94b-kserve-86fcb699c-svmrn" event={"ID":"6a0bd85e-66f4-4942-840a-7f91e0413a8a","Type":"ContainerStarted","Data":"e2b5e75f0c2d37d7227d4b1807225aebb2f464443b4b2dbeeee45faa01d29ec5"}
Oct 30 22:37:03.010725 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:37:03.010700    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-40229e-f6c86585d-gdhkl" event={"ID":"8c927c80-1da0-4b5b-8655-c67e8a35f000","Type":"ContainerStarted","Data":"664b4ceebfaa55cb32f797d5d2b9d53b7fd0eab04588173a390c2d58e450fc47"}
Oct 30 22:37:03.010869 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:37:03.010816    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-40229e-f6c86585d-gdhkl"
Oct 30 22:37:03.051973 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:37:03.051473    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-bdd94b/test-model-bdd94b-kserve-86fcb699c-svmrn" podStartSLOduration=2.051455559 podStartE2EDuration="2.051455559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:37:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:37:03.04960376 +0000 UTC m=+1806.975529994" watchObservedRunningTime="2026-10-30 22:37:03.051455559 +0000 UTC m=+1806.977381765"
Oct 30 22:37:03.090710 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:37:03.090671    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-40229e-f6c86585d-gdhkl" podStartSLOduration=2.73900452 podStartE2EDuration="3.09065945s" totalImagesPullingTime="351.65493ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:37:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:37:03.089080847 +0000 UTC m=+1807.015007050" watchObservedRunningTime="2026-10-30 22:37:03.09065945 +0000 UTC m=+1807.016585656"
Oct 30 22:37:03.648491 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:37:03.648461    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-bdd94b/test-model-bdd94b-kserve-86fcb699c-svmrn"
Oct 30 22:37:09.023522 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:37:09.023487    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-40229e-f6c86585d-gdhkl"
Oct 30 22:37:11.648267 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:37:11.648233    4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-bdd94b/test-model-bdd94b-kserve-86fcb699c-svmrn"
Oct 30 22:37:11.661941 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:37:11.661917    4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-bdd94b/test-model-bdd94b-kserve-86fcb699c-svmrn"
Oct 30 22:37:12.102526 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:37:12.102464    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-bdd94b/test-model-bdd94b-kserve-86fcb699c-svmrn"
Oct 30 22:37:14.532360 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:37:14.532326    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6bz92"]
Oct 30 22:37:26.268548 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:37:26.268506    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 22:37:42.411232 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:37:42.411154    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-bc48cc-6b58bfd8c7-xq5vn"]
Oct 30 22:37:42.419740 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:37:42.419715    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bc48cc-6b58bfd8c7-xq5vn"
Oct 30 22:37:42.438082 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:37:42.438024    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-bc48cc-6b58bfd8c7-xq5vn"]
Oct 30 22:37:42.444338 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:37:42.444306    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-bc48cc-serving-cert\""
Oct 30 22:37:42.531108 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:37:42.531078    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08ebd2bf-5da3-418e-ac08-9584138d0b16-openshift-service-ca\") pod \"maas-api-e2e-inf-a-bc48cc-6b58bfd8c7-xq5vn\" (UID: \"08ebd2bf-5da3-418e-ac08-9584138d0b16\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bc48cc-6b58bfd8c7-xq5vn"
Oct 30 22:37:42.531277 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:37:42.531153    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkbzr\" (UniqueName: \"kubernetes.io/projected/08ebd2bf-5da3-418e-ac08-9584138d0b16-kube-api-access-nkbzr\") pod \"maas-api-e2e-inf-a-bc48cc-6b58bfd8c7-xq5vn\" (UID: \"08ebd2bf-5da3-418e-ac08-9584138d0b16\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bc48cc-6b58bfd8c7-xq5vn"
Oct 30 22:37:42.531382 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:37:42.531359    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08ebd2bf-5da3-418e-ac08-9584138d0b16-maas-api-tls\") pod \"maas-api-e2e-inf-a-bc48cc-6b58bfd8c7-xq5vn\" (UID: \"08ebd2bf-5da3-418e-ac08-9584138d0b16\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bc48cc-6b58bfd8c7-xq5vn"
Oct 30 22:37:42.531420 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:37:42.531407    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08ebd2bf-5da3-418e-ac08-9584138d0b16-metrics-certs\") pod \"maas-api-e2e-inf-a-bc48cc-6b58bfd8c7-xq5vn\" (UID: \"08ebd2bf-5da3-418e-ac08-9584138d0b16\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bc48cc-6b58bfd8c7-xq5vn"
Oct 30 22:37:42.632484 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:37:42.632451    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkbzr\" (UniqueName: \"kubernetes.io/projected/08ebd2bf-5da3-418e-ac08-9584138d0b16-kube-api-access-nkbzr\") pod \"maas-api-e2e-inf-a-bc48cc-6b58bfd8c7-xq5vn\" (UID: \"08ebd2bf-5da3-418e-ac08-9584138d0b16\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bc48cc-6b58bfd8c7-xq5vn"
Oct 30 22:37:42.632645 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:37:42.632597    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08ebd2bf-5da3-418e-ac08-9584138d0b16-maas-api-tls\") pod \"maas-api-e2e-inf-a-bc48cc-6b58bfd8c7-xq5vn\" (UID: \"08ebd2bf-5da3-418e-ac08-9584138d0b16\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bc48cc-6b58bfd8c7-xq5vn"
Oct 30 22:37:42.632645 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:37:42.632634    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08ebd2bf-5da3-418e-ac08-9584138d0b16-metrics-certs\") pod \"maas-api-e2e-inf-a-bc48cc-6b58bfd8c7-xq5vn\" (UID: \"08ebd2bf-5da3-418e-ac08-9584138d0b16\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bc48cc-6b58bfd8c7-xq5vn"
Oct 30 22:37:42.632726 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:37:42.632696    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08ebd2bf-5da3-418e-ac08-9584138d0b16-openshift-service-ca\") pod \"maas-api-e2e-inf-a-bc48cc-6b58bfd8c7-xq5vn\" (UID: \"08ebd2bf-5da3-418e-ac08-9584138d0b16\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bc48cc-6b58bfd8c7-xq5vn"
Oct 30 22:37:42.633369 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:37:42.633338    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08ebd2bf-5da3-418e-ac08-9584138d0b16-openshift-service-ca\") pod \"maas-api-e2e-inf-a-bc48cc-6b58bfd8c7-xq5vn\" (UID: \"08ebd2bf-5da3-418e-ac08-9584138d0b16\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bc48cc-6b58bfd8c7-xq5vn"
Oct 30 22:37:42.635222 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:37:42.635198    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08ebd2bf-5da3-418e-ac08-9584138d0b16-maas-api-tls\") pod \"maas-api-e2e-inf-a-bc48cc-6b58bfd8c7-xq5vn\" (UID: \"08ebd2bf-5da3-418e-ac08-9584138d0b16\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bc48cc-6b58bfd8c7-xq5vn"
Oct 30 22:37:42.635338 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:37:42.635201    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08ebd2bf-5da3-418e-ac08-9584138d0b16-metrics-certs\") pod \"maas-api-e2e-inf-a-bc48cc-6b58bfd8c7-xq5vn\" (UID: \"08ebd2bf-5da3-418e-ac08-9584138d0b16\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bc48cc-6b58bfd8c7-xq5vn"
Oct 30 22:37:42.653762 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:37:42.653736    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkbzr\" (UniqueName: \"kubernetes.io/projected/08ebd2bf-5da3-418e-ac08-9584138d0b16-kube-api-access-nkbzr\") pod \"maas-api-e2e-inf-a-bc48cc-6b58bfd8c7-xq5vn\" (UID: \"08ebd2bf-5da3-418e-ac08-9584138d0b16\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bc48cc-6b58bfd8c7-xq5vn"
Oct 30 22:37:42.740638 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:37:42.740543    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bc48cc-6b58bfd8c7-xq5vn"
Oct 30 22:37:43.063973 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:37:43.063902    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-bc48cc-6b58bfd8c7-xq5vn"]
Oct 30 22:37:43.067606 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:37:43.067574    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08ebd2bf_5da3_418e_ac08_9584138d0b16.slice/crio-b0b374ebae01f3616e35fe031d5ed6b54c277e2fa32d45792fbdc6915c98f0f0 WatchSource:0}: Error finding container b0b374ebae01f3616e35fe031d5ed6b54c277e2fa32d45792fbdc6915c98f0f0: Status 404 returned error can't find the container with id b0b374ebae01f3616e35fe031d5ed6b54c277e2fa32d45792fbdc6915c98f0f0
Oct 30 22:37:43.308874 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:37:43.308834    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bc48cc-6b58bfd8c7-xq5vn" event={"ID":"08ebd2bf-5da3-418e-ac08-9584138d0b16","Type":"ContainerStarted","Data":"b0b374ebae01f3616e35fe031d5ed6b54c277e2fa32d45792fbdc6915c98f0f0"}
Oct 30 22:37:44.322062 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:37:44.321199    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bc48cc-6b58bfd8c7-xq5vn" event={"ID":"08ebd2bf-5da3-418e-ac08-9584138d0b16","Type":"ContainerStarted","Data":"c29009e2f79e779de21ff2c57c6abe7f0ff0033811ca1bfd713f135f157b5691"}
Oct 30 22:37:44.322062 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:37:44.321474    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bc48cc-6b58bfd8c7-xq5vn"
Oct 30 22:37:44.361905 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:37:44.361850    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bc48cc-6b58bfd8c7-xq5vn" podStartSLOduration=2.004158335 podStartE2EDuration="2.361836296s" totalImagesPullingTime="357.677961ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:37:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:37:44.361180368 +0000 UTC m=+1848.287106585" watchObservedRunningTime="2026-10-30 22:37:44.361836296 +0000 UTC m=+1848.287762501"
Oct 30 22:37:49.760072 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:37:49.760022    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-54c7f0-84c875b494-fqqw8"]
Oct 30 22:37:49.764996 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:37:49.764973    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-54c7f0-84c875b494-fqqw8"
Oct 30 22:37:49.779275 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:37:49.779255    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-54c7f0-serving-cert\""
Oct 30 22:37:49.793735 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:37:49.793710    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-54c7f0-84c875b494-fqqw8"]
Oct 30 22:37:49.948718 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:37:49.948679    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b47ab287-9d84-468d-8afb-28b3f6b862e2-maas-api-tls\") pod \"maas-api-e2e-inf-b-54c7f0-84c875b494-fqqw8\" (UID: \"b47ab287-9d84-468d-8afb-28b3f6b862e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-54c7f0-84c875b494-fqqw8"
Oct 30 22:37:49.948879 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:37:49.948744    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnp2t\" (UniqueName: \"kubernetes.io/projected/b47ab287-9d84-468d-8afb-28b3f6b862e2-kube-api-access-tnp2t\") pod \"maas-api-e2e-inf-b-54c7f0-84c875b494-fqqw8\" (UID: \"b47ab287-9d84-468d-8afb-28b3f6b862e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-54c7f0-84c875b494-fqqw8"
Oct 30 22:37:49.948956 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:37:49.948934    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b47ab287-9d84-468d-8afb-28b3f6b862e2-openshift-service-ca\") pod \"maas-api-e2e-inf-b-54c7f0-84c875b494-fqqw8\" (UID: \"b47ab287-9d84-468d-8afb-28b3f6b862e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-54c7f0-84c875b494-fqqw8"
Oct 30 22:37:49.949123 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:37:49.949101    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b47ab287-9d84-468d-8afb-28b3f6b862e2-metrics-certs\") pod \"maas-api-e2e-inf-b-54c7f0-84c875b494-fqqw8\" (UID: \"b47ab287-9d84-468d-8afb-28b3f6b862e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-54c7f0-84c875b494-fqqw8"
Oct 30 22:37:50.051459 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:37:50.050492    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b47ab287-9d84-468d-8afb-28b3f6b862e2-openshift-service-ca\") pod \"maas-api-e2e-inf-b-54c7f0-84c875b494-fqqw8\" (UID: \"b47ab287-9d84-468d-8afb-28b3f6b862e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-54c7f0-84c875b494-fqqw8"
Oct 30 22:37:50.051459 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:37:50.050652    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b47ab287-9d84-468d-8afb-28b3f6b862e2-metrics-certs\") pod \"maas-api-e2e-inf-b-54c7f0-84c875b494-fqqw8\" (UID: \"b47ab287-9d84-468d-8afb-28b3f6b862e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-54c7f0-84c875b494-fqqw8"
Oct 30 22:37:50.051459 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:37:50.050754    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b47ab287-9d84-468d-8afb-28b3f6b862e2-maas-api-tls\") pod \"maas-api-e2e-inf-b-54c7f0-84c875b494-fqqw8\" (UID: \"b47ab287-9d84-468d-8afb-28b3f6b862e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-54c7f0-84c875b494-fqqw8"
Oct 30 22:37:50.051459 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:37:50.050861    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnp2t\" (UniqueName: \"kubernetes.io/projected/b47ab287-9d84-468d-8afb-28b3f6b862e2-kube-api-access-tnp2t\") pod \"maas-api-e2e-inf-b-54c7f0-84c875b494-fqqw8\" (UID: \"b47ab287-9d84-468d-8afb-28b3f6b862e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-54c7f0-84c875b494-fqqw8"
Oct 30 22:37:50.051459 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:37:50.051375    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b47ab287-9d84-468d-8afb-28b3f6b862e2-openshift-service-ca\") pod \"maas-api-e2e-inf-b-54c7f0-84c875b494-fqqw8\" (UID: \"b47ab287-9d84-468d-8afb-28b3f6b862e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-54c7f0-84c875b494-fqqw8"
Oct 30 22:37:50.051855 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:37:50.051643    4477 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-54c7f0-serving-cert: secret "maas-api-e2e-inf-b-54c7f0-serving-cert" not found
Oct 30 22:37:50.051855 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:37:50.051716    4477 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b47ab287-9d84-468d-8afb-28b3f6b862e2-maas-api-tls podName:b47ab287-9d84-468d-8afb-28b3f6b862e2 nodeName:}" failed. No retries permitted until 2026-10-30 22:37:50.55169761 +0000 UTC m=+1854.477623797 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b47ab287-9d84-468d-8afb-28b3f6b862e2-maas-api-tls") pod "maas-api-e2e-inf-b-54c7f0-84c875b494-fqqw8" (UID: "b47ab287-9d84-468d-8afb-28b3f6b862e2") : secret "maas-api-e2e-inf-b-54c7f0-serving-cert" not found
Oct 30 22:37:50.054503 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:37:50.054476    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b47ab287-9d84-468d-8afb-28b3f6b862e2-metrics-certs\") pod \"maas-api-e2e-inf-b-54c7f0-84c875b494-fqqw8\" (UID: \"b47ab287-9d84-468d-8afb-28b3f6b862e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-54c7f0-84c875b494-fqqw8"
Oct 30 22:37:50.070435 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:37:50.070405    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnp2t\" (UniqueName: \"kubernetes.io/projected/b47ab287-9d84-468d-8afb-28b3f6b862e2-kube-api-access-tnp2t\") pod \"maas-api-e2e-inf-b-54c7f0-84c875b494-fqqw8\" (UID: \"b47ab287-9d84-468d-8afb-28b3f6b862e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-54c7f0-84c875b494-fqqw8"
Oct 30 22:37:50.333623 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:37:50.333544    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bc48cc-6b58bfd8c7-xq5vn"
Oct 30 22:37:50.559393 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:37:50.559357    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b47ab287-9d84-468d-8afb-28b3f6b862e2-maas-api-tls\") pod \"maas-api-e2e-inf-b-54c7f0-84c875b494-fqqw8\" (UID: \"b47ab287-9d84-468d-8afb-28b3f6b862e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-54c7f0-84c875b494-fqqw8"
Oct 30 22:37:50.561833 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:37:50.561809    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b47ab287-9d84-468d-8afb-28b3f6b862e2-maas-api-tls\") pod \"maas-api-e2e-inf-b-54c7f0-84c875b494-fqqw8\" (UID: \"b47ab287-9d84-468d-8afb-28b3f6b862e2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-54c7f0-84c875b494-fqqw8"
Oct 30 22:37:50.683101 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:37:50.683082    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-54c7f0-84c875b494-fqqw8"
Oct 30 22:37:50.989923 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:37:50.989856    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-54c7f0-84c875b494-fqqw8"]
Oct 30 22:37:50.994011 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:37:50.993981    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb47ab287_9d84_468d_8afb_28b3f6b862e2.slice/crio-6ec1716f47ec75e0c60dcadb60fc3ae548996d5bb421663383175b67c35d9700 WatchSource:0}: Error finding container 6ec1716f47ec75e0c60dcadb60fc3ae548996d5bb421663383175b67c35d9700: Status 404 returned error can't find the container with id 6ec1716f47ec75e0c60dcadb60fc3ae548996d5bb421663383175b67c35d9700
Oct 30 22:37:51.372663 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:37:51.372573    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-54c7f0-84c875b494-fqqw8" event={"ID":"b47ab287-9d84-468d-8afb-28b3f6b862e2","Type":"ContainerStarted","Data":"6ec1716f47ec75e0c60dcadb60fc3ae548996d5bb421663383175b67c35d9700"}
Oct 30 22:37:52.379947 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:37:52.379907    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-54c7f0-84c875b494-fqqw8" event={"ID":"b47ab287-9d84-468d-8afb-28b3f6b862e2","Type":"ContainerStarted","Data":"d39669020ab4c553387b31855567432e44bfda069c766dfcb3e003361bfc2c05"}
Oct 30 22:37:52.380458 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:37:52.380160    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-54c7f0-84c875b494-fqqw8"
Oct 30 22:37:52.418631 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:37:52.418585    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-54c7f0-84c875b494-fqqw8" podStartSLOduration=3.022051002 podStartE2EDuration="3.418572835s" totalImagesPullingTime="396.521833ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:37:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:37:52.416827131 +0000 UTC m=+1856.342753336" watchObservedRunningTime="2026-10-30 22:37:52.418572835 +0000 UTC m=+1856.344499091"
Oct 30 22:37:55.786170 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:37:55.786130    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-n6fb8"]
Oct 30 22:37:55.786682 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:37:55.786349    4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-n6fb8" podUID="96dc25ab-312f-4915-80ad-35c864d7dfec" containerName="istio-proxy" containerID="cri-o://29833178f4731ebe85ef3a66f4fb3eda5ef0bdaad96caf106b833caebb7813ae" gracePeriod=30
Oct 30 22:37:56.268639 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:37:56.268520    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 22:37:58.394531 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:37:58.394501    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-54c7f0-84c875b494-fqqw8"
Oct 30 22:38:01.112398 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:01.112371    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-n6fb8"
Oct 30 22:38:01.242950 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:01.242918    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96dc25ab-312f-4915-80ad-35c864d7dfec-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96dc25ab-312f-4915-80ad-35c864d7dfec-istio-data\") pod \"96dc25ab-312f-4915-80ad-35c864d7dfec\" (UID: \"96dc25ab-312f-4915-80ad-35c864d7dfec\") "
Oct 30 22:38:01.243161 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:01.242961    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96dc25ab-312f-4915-80ad-35c864d7dfec-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96dc25ab-312f-4915-80ad-35c864d7dfec-istio-envoy\") pod \"96dc25ab-312f-4915-80ad-35c864d7dfec\" (UID: \"96dc25ab-312f-4915-80ad-35c864d7dfec\") "
Oct 30 22:38:01.243161 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:01.242980    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/96dc25ab-312f-4915-80ad-35c864d7dfec-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96dc25ab-312f-4915-80ad-35c864d7dfec-istio-podinfo\") pod \"96dc25ab-312f-4915-80ad-35c864d7dfec\" (UID: \"96dc25ab-312f-4915-80ad-35c864d7dfec\") "
Oct 30 22:38:01.243161 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:01.243024    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96dc25ab-312f-4915-80ad-35c864d7dfec-istio-token\" (UniqueName: \"kubernetes.io/projected/96dc25ab-312f-4915-80ad-35c864d7dfec-istio-token\") pod \"96dc25ab-312f-4915-80ad-35c864d7dfec\" (UID: \"96dc25ab-312f-4915-80ad-35c864d7dfec\") "
Oct 30 22:38:01.243161 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:01.243099    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96dc25ab-312f-4915-80ad-35c864d7dfec-kube-api-access-nlrn5\" (UniqueName: \"kubernetes.io/projected/96dc25ab-312f-4915-80ad-35c864d7dfec-kube-api-access-nlrn5\") pod \"96dc25ab-312f-4915-80ad-35c864d7dfec\" (UID: \"96dc25ab-312f-4915-80ad-35c864d7dfec\") "
Oct 30 22:38:01.243440 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:01.243218    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96dc25ab-312f-4915-80ad-35c864d7dfec-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96dc25ab-312f-4915-80ad-35c864d7dfec-credential-socket\") pod \"96dc25ab-312f-4915-80ad-35c864d7dfec\" (UID: \"96dc25ab-312f-4915-80ad-35c864d7dfec\") "
Oct 30 22:38:01.243440 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:01.243305    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96dc25ab-312f-4915-80ad-35c864d7dfec-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96dc25ab-312f-4915-80ad-35c864d7dfec-workload-certs\") pod \"96dc25ab-312f-4915-80ad-35c864d7dfec\" (UID: \"96dc25ab-312f-4915-80ad-35c864d7dfec\") "
Oct 30 22:38:01.243440 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:01.243434    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96dc25ab-312f-4915-80ad-35c864d7dfec-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96dc25ab-312f-4915-80ad-35c864d7dfec-istiod-ca-cert\") pod \"96dc25ab-312f-4915-80ad-35c864d7dfec\" (UID: \"96dc25ab-312f-4915-80ad-35c864d7dfec\") "
Oct 30 22:38:01.243603 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:01.243444    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96dc25ab-312f-4915-80ad-35c864d7dfec-istio-data" pod "96dc25ab-312f-4915-80ad-35c864d7dfec" (UID: "96dc25ab-312f-4915-80ad-35c864d7dfec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:38:01.243603 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:01.243482    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96dc25ab-312f-4915-80ad-35c864d7dfec-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96dc25ab-312f-4915-80ad-35c864d7dfec-workload-socket\") pod \"96dc25ab-312f-4915-80ad-35c864d7dfec\" (UID: \"96dc25ab-312f-4915-80ad-35c864d7dfec\") "
Oct 30 22:38:01.243918 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:01.243815    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96dc25ab-312f-4915-80ad-35c864d7dfec-istiod-ca-cert" pod "96dc25ab-312f-4915-80ad-35c864d7dfec" (UID: "96dc25ab-312f-4915-80ad-35c864d7dfec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:38:01.244135 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:01.243888    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96dc25ab-312f-4915-80ad-35c864d7dfec-workload-socket" pod "96dc25ab-312f-4915-80ad-35c864d7dfec" (UID: "96dc25ab-312f-4915-80ad-35c864d7dfec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:38:01.244351 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:01.244286    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96dc25ab-312f-4915-80ad-35c864d7dfec-workload-certs" pod "96dc25ab-312f-4915-80ad-35c864d7dfec" (UID: "96dc25ab-312f-4915-80ad-35c864d7dfec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:38:01.244453 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:01.244339    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96dc25ab-312f-4915-80ad-35c864d7dfec-credential-socket" pod "96dc25ab-312f-4915-80ad-35c864d7dfec" (UID: "96dc25ab-312f-4915-80ad-35c864d7dfec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:38:01.244453 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:01.244261    4477 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96dc25ab-312f-4915-80ad-35c864d7dfec-istiod-ca-cert\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:38:01.244453 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:01.244422    4477 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96dc25ab-312f-4915-80ad-35c864d7dfec-istio-data\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:38:01.245862 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:01.245823    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96dc25ab-312f-4915-80ad-35c864d7dfec-istio-token" pod "96dc25ab-312f-4915-80ad-35c864d7dfec" (UID: "96dc25ab-312f-4915-80ad-35c864d7dfec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:38:01.246092 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:01.246032    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96dc25ab-312f-4915-80ad-35c864d7dfec-istio-envoy" pod "96dc25ab-312f-4915-80ad-35c864d7dfec" (UID: "96dc25ab-312f-4915-80ad-35c864d7dfec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:38:01.246285 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:01.246260    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96dc25ab-312f-4915-80ad-35c864d7dfec-kube-api-access-nlrn5" pod "96dc25ab-312f-4915-80ad-35c864d7dfec" (UID: "96dc25ab-312f-4915-80ad-35c864d7dfec"). InnerVolumeSpecName "kube-api-access-nlrn5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:38:01.246587 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:01.246558    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/96dc25ab-312f-4915-80ad-35c864d7dfec-istio-podinfo" pod "96dc25ab-312f-4915-80ad-35c864d7dfec" (UID: "96dc25ab-312f-4915-80ad-35c864d7dfec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:38:01.346137 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:01.346056    4477 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96dc25ab-312f-4915-80ad-35c864d7dfec-istio-envoy\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:38:01.346137 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:01.346088    4477 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96dc25ab-312f-4915-80ad-35c864d7dfec-istio-podinfo\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:38:01.346137 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:01.346100    4477 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96dc25ab-312f-4915-80ad-35c864d7dfec-istio-token\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:38:01.346137 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:01.346115    4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlrn5\" (UniqueName: \"kubernetes.io/projected/96dc25ab-312f-4915-80ad-35c864d7dfec-kube-api-access-nlrn5\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:38:01.346137 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:01.346128    4477 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96dc25ab-312f-4915-80ad-35c864d7dfec-credential-socket\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:38:01.346137 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:01.346142    4477 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96dc25ab-312f-4915-80ad-35c864d7dfec-workload-certs\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:38:01.346414 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:01.346155    4477 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96dc25ab-312f-4915-80ad-35c864d7dfec-workload-socket\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:38:01.436117 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:01.436087    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6e0edc-6fb6b889ff-96k9w"]
Oct 30 22:38:01.436752 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:01.436734    4477 cpu_manager.go:414] "containerMap: removing container" podUID="96dc25ab-312f-4915-80ad-35c864d7dfec" containerName="istio-proxy"
Oct 30 22:38:01.436801 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:01.436753    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96dc25ab-312f-4915-80ad-35c864d7dfec" containerName="istio-proxy"
Oct 30 22:38:01.436884 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:01.436875    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="96dc25ab-312f-4915-80ad-35c864d7dfec" containerName="istio-proxy"
Oct 30 22:38:01.441210 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:01.441190    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6e0edc-6fb6b889ff-96k9w"
Oct 30 22:38:01.445580 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:01.445554    4477 generic.go:356] "Generic (PLEG): container finished" podID="96dc25ab-312f-4915-80ad-35c864d7dfec" containerID="29833178f4731ebe85ef3a66f4fb3eda5ef0bdaad96caf106b833caebb7813ae" exitCode=0
Oct 30 22:38:01.445690 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:01.445599    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-n6fb8"
Oct 30 22:38:01.445690 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:01.445673    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-n6fb8" event={"ID":"96dc25ab-312f-4915-80ad-35c864d7dfec","Type":"ContainerDied","Data":"29833178f4731ebe85ef3a66f4fb3eda5ef0bdaad96caf106b833caebb7813ae"}
Oct 30 22:38:01.445782 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:01.445724    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-n6fb8" event={"ID":"96dc25ab-312f-4915-80ad-35c864d7dfec","Type":"ContainerDied","Data":"91c1634190b110e4e6c4573370ac69d5f7d4daf0e69e4ab1758bd4d0f31804d2"}
Oct 30 22:38:01.445782 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:01.445747    4477 scope.go:122] "RemoveContainer" containerID="29833178f4731ebe85ef3a66f4fb3eda5ef0bdaad96caf106b833caebb7813ae"
Oct 30 22:38:01.458814 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:01.458718    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-6e0edc-serving-cert\""
Oct 30 22:38:01.459000 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:01.458979    4477 scope.go:122] "RemoveContainer" containerID="29833178f4731ebe85ef3a66f4fb3eda5ef0bdaad96caf106b833caebb7813ae"
Oct 30 22:38:01.459524 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:38:01.459442    4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29833178f4731ebe85ef3a66f4fb3eda5ef0bdaad96caf106b833caebb7813ae\": container with ID starting with 29833178f4731ebe85ef3a66f4fb3eda5ef0bdaad96caf106b833caebb7813ae not found: ID does not exist" containerID="29833178f4731ebe85ef3a66f4fb3eda5ef0bdaad96caf106b833caebb7813ae"
Oct 30 22:38:01.459524 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:01.459476    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29833178f4731ebe85ef3a66f4fb3eda5ef0bdaad96caf106b833caebb7813ae"} err="failed to get container status \"29833178f4731ebe85ef3a66f4fb3eda5ef0bdaad96caf106b833caebb7813ae\": rpc error: code = NotFound desc = could not find container \"29833178f4731ebe85ef3a66f4fb3eda5ef0bdaad96caf106b833caebb7813ae\": container with ID starting with 29833178f4731ebe85ef3a66f4fb3eda5ef0bdaad96caf106b833caebb7813ae not found: ID does not exist"
Oct 30 22:38:01.461336 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:01.461317    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6e0edc-6fb6b889ff-96k9w"]
Oct 30 22:38:01.514307 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:01.514246    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-n6fb8"]
Oct 30 22:38:01.518884 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:01.518852    4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-n6fb8"]
Oct 30 22:38:01.548261 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:01.548229    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6d933b6-d604-4487-aad9-41feae842796-metrics-certs\") pod \"maas-api-e2e-inf-a-6e0edc-6fb6b889ff-96k9w\" (UID: \"d6d933b6-d604-4487-aad9-41feae842796\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6e0edc-6fb6b889ff-96k9w"
Oct 30 22:38:01.548413 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:01.548391    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6d933b6-d604-4487-aad9-41feae842796-openshift-service-ca\") pod \"maas-api-e2e-inf-a-6e0edc-6fb6b889ff-96k9w\" (UID: \"d6d933b6-d604-4487-aad9-41feae842796\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6e0edc-6fb6b889ff-96k9w"
Oct 30 22:38:01.548458 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:01.548448    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6d933b6-d604-4487-aad9-41feae842796-maas-api-tls\") pod \"maas-api-e2e-inf-a-6e0edc-6fb6b889ff-96k9w\" (UID: \"d6d933b6-d604-4487-aad9-41feae842796\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6e0edc-6fb6b889ff-96k9w"
Oct 30 22:38:01.548643 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:01.548625    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzqx9\" (UniqueName: \"kubernetes.io/projected/d6d933b6-d604-4487-aad9-41feae842796-kube-api-access-nzqx9\") pod \"maas-api-e2e-inf-a-6e0edc-6fb6b889ff-96k9w\" (UID: \"d6d933b6-d604-4487-aad9-41feae842796\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6e0edc-6fb6b889ff-96k9w"
Oct 30 22:38:01.650302 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:01.650269    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6d933b6-d604-4487-aad9-41feae842796-maas-api-tls\") pod \"maas-api-e2e-inf-a-6e0edc-6fb6b889ff-96k9w\" (UID: \"d6d933b6-d604-4487-aad9-41feae842796\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6e0edc-6fb6b889ff-96k9w"
Oct 30 22:38:01.650511 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:01.650377    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzqx9\" (UniqueName: \"kubernetes.io/projected/d6d933b6-d604-4487-aad9-41feae842796-kube-api-access-nzqx9\") pod \"maas-api-e2e-inf-a-6e0edc-6fb6b889ff-96k9w\" (UID: \"d6d933b6-d604-4487-aad9-41feae842796\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6e0edc-6fb6b889ff-96k9w"
Oct 30 22:38:01.650511 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:38:01.650440    4477 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-6e0edc-serving-cert: secret "maas-api-e2e-inf-a-6e0edc-serving-cert" not found
Oct 30 22:38:01.650511 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:01.650457    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6d933b6-d604-4487-aad9-41feae842796-metrics-certs\") pod \"maas-api-e2e-inf-a-6e0edc-6fb6b889ff-96k9w\" (UID: \"d6d933b6-d604-4487-aad9-41feae842796\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6e0edc-6fb6b889ff-96k9w"
Oct 30 22:38:01.650511 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:38:01.650511    4477 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6d933b6-d604-4487-aad9-41feae842796-maas-api-tls podName:d6d933b6-d604-4487-aad9-41feae842796 nodeName:}" failed. No retries permitted until 2026-10-30 22:38:02.150491005 +0000 UTC m=+1866.076417192 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d6d933b6-d604-4487-aad9-41feae842796-maas-api-tls") pod "maas-api-e2e-inf-a-6e0edc-6fb6b889ff-96k9w" (UID: "d6d933b6-d604-4487-aad9-41feae842796") : secret "maas-api-e2e-inf-a-6e0edc-serving-cert" not found
Oct 30 22:38:01.650819 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:01.650653    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6d933b6-d604-4487-aad9-41feae842796-openshift-service-ca\") pod \"maas-api-e2e-inf-a-6e0edc-6fb6b889ff-96k9w\" (UID: \"d6d933b6-d604-4487-aad9-41feae842796\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6e0edc-6fb6b889ff-96k9w"
Oct 30 22:38:01.651445 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:01.651420    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6d933b6-d604-4487-aad9-41feae842796-openshift-service-ca\") pod \"maas-api-e2e-inf-a-6e0edc-6fb6b889ff-96k9w\" (UID: \"d6d933b6-d604-4487-aad9-41feae842796\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6e0edc-6fb6b889ff-96k9w"
Oct 30 22:38:01.653978 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:01.653954    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6d933b6-d604-4487-aad9-41feae842796-metrics-certs\") pod \"maas-api-e2e-inf-a-6e0edc-6fb6b889ff-96k9w\" (UID: \"d6d933b6-d604-4487-aad9-41feae842796\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6e0edc-6fb6b889ff-96k9w"
Oct 30 22:38:01.675066 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:01.673463    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzqx9\" (UniqueName: \"kubernetes.io/projected/d6d933b6-d604-4487-aad9-41feae842796-kube-api-access-nzqx9\") pod \"maas-api-e2e-inf-a-6e0edc-6fb6b889ff-96k9w\" (UID: \"d6d933b6-d604-4487-aad9-41feae842796\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6e0edc-6fb6b889ff-96k9w"
Oct 30 22:38:02.158858 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:02.158819    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6d933b6-d604-4487-aad9-41feae842796-maas-api-tls\") pod \"maas-api-e2e-inf-a-6e0edc-6fb6b889ff-96k9w\" (UID: \"d6d933b6-d604-4487-aad9-41feae842796\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6e0edc-6fb6b889ff-96k9w"
Oct 30 22:38:02.161293 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:02.161272    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6d933b6-d604-4487-aad9-41feae842796-maas-api-tls\") pod \"maas-api-e2e-inf-a-6e0edc-6fb6b889ff-96k9w\" (UID: \"d6d933b6-d604-4487-aad9-41feae842796\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6e0edc-6fb6b889ff-96k9w"
Oct 30 22:38:02.197374 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:02.197339    4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96dc25ab-312f-4915-80ad-35c864d7dfec" path="/var/lib/kubelet/pods/96dc25ab-312f-4915-80ad-35c864d7dfec/volumes"
Oct 30 22:38:02.362979 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:02.362946    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6e0edc-6fb6b889ff-96k9w"
Oct 30 22:38:02.721966 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:02.721935    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6e0edc-6fb6b889ff-96k9w"]
Oct 30 22:38:02.725533 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:38:02.725503    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6d933b6_d604_4487_aad9_41feae842796.slice/crio-1bee247c54e880cf4e0081ef0b05cd7461715ef66fdd085f914a566fded3a17a WatchSource:0}: Error finding container 1bee247c54e880cf4e0081ef0b05cd7461715ef66fdd085f914a566fded3a17a: Status 404 returned error can't find the container with id 1bee247c54e880cf4e0081ef0b05cd7461715ef66fdd085f914a566fded3a17a
Oct 30 22:38:03.470948 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:03.470907    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6e0edc-6fb6b889ff-96k9w" event={"ID":"d6d933b6-d604-4487-aad9-41feae842796","Type":"ContainerStarted","Data":"0216c55d852e529e1b352b20ca4a09ae2d0e38faf635b5152bf661dee93275c9"}
Oct 30 22:38:03.471393 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:03.470978    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6e0edc-6fb6b889ff-96k9w" event={"ID":"d6d933b6-d604-4487-aad9-41feae842796","Type":"ContainerStarted","Data":"1bee247c54e880cf4e0081ef0b05cd7461715ef66fdd085f914a566fded3a17a"}
Oct 30 22:38:03.471393 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:03.471015    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6e0edc-6fb6b889ff-96k9w"
Oct 30 22:38:03.513020 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:03.512970    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6e0edc-6fb6b889ff-96k9w" podStartSLOduration=2.225642416 podStartE2EDuration="2.512955452s" totalImagesPullingTime="287.313036ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:38:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:38:03.508686042 +0000 UTC m=+1867.434612265" watchObservedRunningTime="2026-10-30 22:38:03.512955452 +0000 UTC m=+1867.438881658"
Oct 30 22:38:09.483804 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:09.483773    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6e0edc-6fb6b889ff-96k9w"
Oct 30 22:38:16.261866 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:16.261826    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f0bb45-fbdcb4dd6-pvjvt"]
Oct 30 22:38:16.266537 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:16.266512    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f0bb45-fbdcb4dd6-pvjvt"
Oct 30 22:38:16.283371 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:16.283350    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-f0bb45-serving-cert\""
Oct 30 22:38:16.295011 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:16.294983    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f0bb45-fbdcb4dd6-pvjvt"]
Oct 30 22:38:16.376363 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:16.376331    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/616d47a3-a660-48d3-9714-9e0f45a5660a-metrics-certs\") pod \"maas-api-e2e-inf-b-f0bb45-fbdcb4dd6-pvjvt\" (UID: \"616d47a3-a660-48d3-9714-9e0f45a5660a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f0bb45-fbdcb4dd6-pvjvt"
Oct 30 22:38:16.376512 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:16.376436    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/616d47a3-a660-48d3-9714-9e0f45a5660a-openshift-service-ca\") pod \"maas-api-e2e-inf-b-f0bb45-fbdcb4dd6-pvjvt\" (UID: \"616d47a3-a660-48d3-9714-9e0f45a5660a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f0bb45-fbdcb4dd6-pvjvt"
Oct 30 22:38:16.376584 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:16.376563    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/616d47a3-a660-48d3-9714-9e0f45a5660a-maas-api-tls\") pod \"maas-api-e2e-inf-b-f0bb45-fbdcb4dd6-pvjvt\" (UID: \"616d47a3-a660-48d3-9714-9e0f45a5660a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f0bb45-fbdcb4dd6-pvjvt"
Oct 30 22:38:16.376665 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:16.376651    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szwtw\" (UniqueName: \"kubernetes.io/projected/616d47a3-a660-48d3-9714-9e0f45a5660a-kube-api-access-szwtw\") pod \"maas-api-e2e-inf-b-f0bb45-fbdcb4dd6-pvjvt\" (UID: \"616d47a3-a660-48d3-9714-9e0f45a5660a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f0bb45-fbdcb4dd6-pvjvt"
Oct 30 22:38:16.478233 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:16.478194    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/616d47a3-a660-48d3-9714-9e0f45a5660a-maas-api-tls\") pod \"maas-api-e2e-inf-b-f0bb45-fbdcb4dd6-pvjvt\" (UID: \"616d47a3-a660-48d3-9714-9e0f45a5660a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f0bb45-fbdcb4dd6-pvjvt"
Oct 30 22:38:16.478415 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:16.478285    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szwtw\" (UniqueName: \"kubernetes.io/projected/616d47a3-a660-48d3-9714-9e0f45a5660a-kube-api-access-szwtw\") pod \"maas-api-e2e-inf-b-f0bb45-fbdcb4dd6-pvjvt\" (UID: \"616d47a3-a660-48d3-9714-9e0f45a5660a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f0bb45-fbdcb4dd6-pvjvt"
Oct 30 22:38:16.478488 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:16.478474    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/616d47a3-a660-48d3-9714-9e0f45a5660a-metrics-certs\") pod \"maas-api-e2e-inf-b-f0bb45-fbdcb4dd6-pvjvt\" (UID: \"616d47a3-a660-48d3-9714-9e0f45a5660a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f0bb45-fbdcb4dd6-pvjvt"
Oct 30 22:38:16.478531 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:16.478514    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/616d47a3-a660-48d3-9714-9e0f45a5660a-openshift-service-ca\") pod \"maas-api-e2e-inf-b-f0bb45-fbdcb4dd6-pvjvt\" (UID: \"616d47a3-a660-48d3-9714-9e0f45a5660a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f0bb45-fbdcb4dd6-pvjvt"
Oct 30 22:38:16.479250 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:16.479223    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/616d47a3-a660-48d3-9714-9e0f45a5660a-openshift-service-ca\") pod \"maas-api-e2e-inf-b-f0bb45-fbdcb4dd6-pvjvt\" (UID: \"616d47a3-a660-48d3-9714-9e0f45a5660a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f0bb45-fbdcb4dd6-pvjvt"
Oct 30 22:38:16.482080 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:16.482056    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/616d47a3-a660-48d3-9714-9e0f45a5660a-metrics-certs\") pod \"maas-api-e2e-inf-b-f0bb45-fbdcb4dd6-pvjvt\" (UID: \"616d47a3-a660-48d3-9714-9e0f45a5660a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f0bb45-fbdcb4dd6-pvjvt"
Oct 30 22:38:16.482324 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:16.482059    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/616d47a3-a660-48d3-9714-9e0f45a5660a-maas-api-tls\") pod \"maas-api-e2e-inf-b-f0bb45-fbdcb4dd6-pvjvt\" (UID: \"616d47a3-a660-48d3-9714-9e0f45a5660a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f0bb45-fbdcb4dd6-pvjvt"
Oct 30 22:38:16.497935 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:16.497905    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szwtw\" (UniqueName: \"kubernetes.io/projected/616d47a3-a660-48d3-9714-9e0f45a5660a-kube-api-access-szwtw\") pod \"maas-api-e2e-inf-b-f0bb45-fbdcb4dd6-pvjvt\" (UID: \"616d47a3-a660-48d3-9714-9e0f45a5660a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f0bb45-fbdcb4dd6-pvjvt"
Oct 30 22:38:16.580430 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:16.580354    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f0bb45-fbdcb4dd6-pvjvt"
Oct 30 22:38:16.882009 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:16.881980    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f0bb45-fbdcb4dd6-pvjvt"]
Oct 30 22:38:16.884789 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:38:16.884751    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod616d47a3_a660_48d3_9714_9e0f45a5660a.slice/crio-cf628afec8a1c2982ed7def8a7d8f76522cd41efb3e97d33e7203a89cc59f2a7 WatchSource:0}: Error finding container cf628afec8a1c2982ed7def8a7d8f76522cd41efb3e97d33e7203a89cc59f2a7: Status 404 returned error can't find the container with id cf628afec8a1c2982ed7def8a7d8f76522cd41efb3e97d33e7203a89cc59f2a7
Oct 30 22:38:17.681570 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:17.679736    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f0bb45-fbdcb4dd6-pvjvt" event={"ID":"616d47a3-a660-48d3-9714-9e0f45a5660a","Type":"ContainerStarted","Data":"3b091c37446e46d81a03d352931431ae28cf96b5997c11435a783b49bc2ddee4"}
Oct 30 22:38:17.681570 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:17.679832    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f0bb45-fbdcb4dd6-pvjvt" event={"ID":"616d47a3-a660-48d3-9714-9e0f45a5660a","Type":"ContainerStarted","Data":"cf628afec8a1c2982ed7def8a7d8f76522cd41efb3e97d33e7203a89cc59f2a7"}
Oct 30 22:38:17.681570 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:17.680909    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f0bb45-fbdcb4dd6-pvjvt"
Oct 30 22:38:17.721196 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:17.721138    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f0bb45-fbdcb4dd6-pvjvt" podStartSLOduration=1.361362476 podStartE2EDuration="1.721122976s" totalImagesPullingTime="359.7605ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:38:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:38:17.717292233 +0000 UTC m=+1881.643218463" watchObservedRunningTime="2026-10-30 22:38:17.721122976 +0000 UTC m=+1881.647049181"
Oct 30 22:38:24.698783 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:24.698756    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f0bb45-fbdcb4dd6-pvjvt"
Oct 30 22:38:26.267730 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:38:26.267688    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 22:38:45.767905 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:45.767865    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-999470/test-model-999470-kserve-56bbd9b675-5xsxl"]
Oct 30 22:38:45.777480 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:45.777424    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-999470/test-model-999470-kserve-56bbd9b675-5xsxl"
Oct 30 22:38:45.841881 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:45.841253    4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-999470\"/\"kube-root-ca.crt\""
Oct 30 22:38:45.841881 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:45.841320    4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-999470\"/\"openshift-service-ca.crt\""
Oct 30 22:38:45.841881 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:45.841467    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-999470\"/\"default-dockercfg-9fttj\""
Oct 30 22:38:45.842178 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:45.841907    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-999470\"/\"test-model-999470-kserve-self-signed-certs\""
Oct 30 22:38:45.854801 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:45.854766    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-999470/test-model-999470-kserve-56bbd9b675-5xsxl"]
Oct 30 22:38:45.929289 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:45.929253    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f3b16db-55fe-4505-9f9f-13bbfe486000-tls-certs\") pod \"test-model-999470-kserve-56bbd9b675-5xsxl\" (UID: \"3f3b16db-55fe-4505-9f9f-13bbfe486000\") " pod="ai-tenant-e2e-inf-a-999470/test-model-999470-kserve-56bbd9b675-5xsxl"
Oct 30 22:38:45.929458 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:45.929382    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f3b16db-55fe-4505-9f9f-13bbfe486000-dshm\") pod \"test-model-999470-kserve-56bbd9b675-5xsxl\" (UID: \"3f3b16db-55fe-4505-9f9f-13bbfe486000\") " pod="ai-tenant-e2e-inf-a-999470/test-model-999470-kserve-56bbd9b675-5xsxl"
Oct 30 22:38:45.929582 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:45.929559    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjfsx\" (UniqueName: \"kubernetes.io/projected/3f3b16db-55fe-4505-9f9f-13bbfe486000-kube-api-access-bjfsx\") pod \"test-model-999470-kserve-56bbd9b675-5xsxl\" (UID: \"3f3b16db-55fe-4505-9f9f-13bbfe486000\") " pod="ai-tenant-e2e-inf-a-999470/test-model-999470-kserve-56bbd9b675-5xsxl"
Oct 30 22:38:45.929621 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:45.929606    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f3b16db-55fe-4505-9f9f-13bbfe486000-home\") pod \"test-model-999470-kserve-56bbd9b675-5xsxl\" (UID: \"3f3b16db-55fe-4505-9f9f-13bbfe486000\") " pod="ai-tenant-e2e-inf-a-999470/test-model-999470-kserve-56bbd9b675-5xsxl"
Oct 30 22:38:45.929660 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:45.929626    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f3b16db-55fe-4505-9f9f-13bbfe486000-model-cache\") pod \"test-model-999470-kserve-56bbd9b675-5xsxl\" (UID: \"3f3b16db-55fe-4505-9f9f-13bbfe486000\") " pod="ai-tenant-e2e-inf-a-999470/test-model-999470-kserve-56bbd9b675-5xsxl"
Oct 30 22:38:45.929761 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:45.929749    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f3b16db-55fe-4505-9f9f-13bbfe486000-tmp-dir\") pod \"test-model-999470-kserve-56bbd9b675-5xsxl\" (UID: \"3f3b16db-55fe-4505-9f9f-13bbfe486000\") " pod="ai-tenant-e2e-inf-a-999470/test-model-999470-kserve-56bbd9b675-5xsxl"
Oct 30 22:38:46.032242 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:46.032151    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f3b16db-55fe-4505-9f9f-13bbfe486000-dshm\") pod \"test-model-999470-kserve-56bbd9b675-5xsxl\" (UID: \"3f3b16db-55fe-4505-9f9f-13bbfe486000\") " pod="ai-tenant-e2e-inf-a-999470/test-model-999470-kserve-56bbd9b675-5xsxl"
Oct 30 22:38:46.033310 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:46.033283    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjfsx\" (UniqueName: \"kubernetes.io/projected/3f3b16db-55fe-4505-9f9f-13bbfe486000-kube-api-access-bjfsx\") pod \"test-model-999470-kserve-56bbd9b675-5xsxl\" (UID: \"3f3b16db-55fe-4505-9f9f-13bbfe486000\") " pod="ai-tenant-e2e-inf-a-999470/test-model-999470-kserve-56bbd9b675-5xsxl"
Oct 30 22:38:46.033529 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:46.033511    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f3b16db-55fe-4505-9f9f-13bbfe486000-home\") pod \"test-model-999470-kserve-56bbd9b675-5xsxl\" (UID: \"3f3b16db-55fe-4505-9f9f-13bbfe486000\") " pod="ai-tenant-e2e-inf-a-999470/test-model-999470-kserve-56bbd9b675-5xsxl"
Oct 30 22:38:46.033643 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:46.033629    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f3b16db-55fe-4505-9f9f-13bbfe486000-model-cache\") pod \"test-model-999470-kserve-56bbd9b675-5xsxl\" (UID: \"3f3b16db-55fe-4505-9f9f-13bbfe486000\") " pod="ai-tenant-e2e-inf-a-999470/test-model-999470-kserve-56bbd9b675-5xsxl"
Oct 30 22:38:46.033824 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:46.033809    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f3b16db-55fe-4505-9f9f-13bbfe486000-tmp-dir\") pod \"test-model-999470-kserve-56bbd9b675-5xsxl\" (UID: \"3f3b16db-55fe-4505-9f9f-13bbfe486000\") " pod="ai-tenant-e2e-inf-a-999470/test-model-999470-kserve-56bbd9b675-5xsxl"
Oct 30 22:38:46.034060 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:46.034028    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f3b16db-55fe-4505-9f9f-13bbfe486000-tls-certs\") pod \"test-model-999470-kserve-56bbd9b675-5xsxl\" (UID: \"3f3b16db-55fe-4505-9f9f-13bbfe486000\") " pod="ai-tenant-e2e-inf-a-999470/test-model-999470-kserve-56bbd9b675-5xsxl"
Oct 30 22:38:46.034918 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:46.034893    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f3b16db-55fe-4505-9f9f-13bbfe486000-home\") pod \"test-model-999470-kserve-56bbd9b675-5xsxl\" (UID: \"3f3b16db-55fe-4505-9f9f-13bbfe486000\") " pod="ai-tenant-e2e-inf-a-999470/test-model-999470-kserve-56bbd9b675-5xsxl"
Oct 30 22:38:46.052063 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:46.035607    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f3b16db-55fe-4505-9f9f-13bbfe486000-model-cache\") pod \"test-model-999470-kserve-56bbd9b675-5xsxl\" (UID: \"3f3b16db-55fe-4505-9f9f-13bbfe486000\") " pod="ai-tenant-e2e-inf-a-999470/test-model-999470-kserve-56bbd9b675-5xsxl"
Oct 30 22:38:46.052503 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:46.038536    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f3b16db-55fe-4505-9f9f-13bbfe486000-tmp-dir\") pod \"test-model-999470-kserve-56bbd9b675-5xsxl\" (UID: \"3f3b16db-55fe-4505-9f9f-13bbfe486000\") " pod="ai-tenant-e2e-inf-a-999470/test-model-999470-kserve-56bbd9b675-5xsxl"
Oct 30 22:38:46.069017 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:46.068839    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f3b16db-55fe-4505-9f9f-13bbfe486000-tls-certs\") pod \"test-model-999470-kserve-56bbd9b675-5xsxl\" (UID: \"3f3b16db-55fe-4505-9f9f-13bbfe486000\") " pod="ai-tenant-e2e-inf-a-999470/test-model-999470-kserve-56bbd9b675-5xsxl"
Oct 30 22:38:46.077336 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:46.077276    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjfsx\" (UniqueName: \"kubernetes.io/projected/3f3b16db-55fe-4505-9f9f-13bbfe486000-kube-api-access-bjfsx\") pod \"test-model-999470-kserve-56bbd9b675-5xsxl\" (UID: \"3f3b16db-55fe-4505-9f9f-13bbfe486000\") " pod="ai-tenant-e2e-inf-a-999470/test-model-999470-kserve-56bbd9b675-5xsxl"
Oct 30 22:38:46.088108 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:46.083303    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f3b16db-55fe-4505-9f9f-13bbfe486000-dshm\") pod \"test-model-999470-kserve-56bbd9b675-5xsxl\" (UID: \"3f3b16db-55fe-4505-9f9f-13bbfe486000\") " pod="ai-tenant-e2e-inf-a-999470/test-model-999470-kserve-56bbd9b675-5xsxl"
Oct 30 22:38:46.140567 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:46.140277    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-999470/test-model-999470-kserve-56bbd9b675-5xsxl"
Oct 30 22:38:46.500513 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:46.500488    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-999470/test-model-999470-kserve-56bbd9b675-5xsxl"]
Oct 30 22:38:46.502877 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:38:46.502850    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3f3b16db_55fe_4505_9f9f_13bbfe486000.slice/crio-31d155fa6a70dbc1481d5f57008ef9ba6034ef1a50c22d33253fd32df44fbbeb WatchSource:0}: Error finding container 31d155fa6a70dbc1481d5f57008ef9ba6034ef1a50c22d33253fd32df44fbbeb: Status 404 returned error can't find the container with id 31d155fa6a70dbc1481d5f57008ef9ba6034ef1a50c22d33253fd32df44fbbeb
Oct 30 22:38:46.915999 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:46.915965    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-999470/test-model-999470-kserve-56bbd9b675-5xsxl" event={"ID":"3f3b16db-55fe-4505-9f9f-13bbfe486000","Type":"ContainerStarted","Data":"6482876f76933143cb8378c3777073b4ba5244c0f5b595aa7e1565ba79eccd4c"}
Oct 30 22:38:46.916490 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:46.916005    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-999470/test-model-999470-kserve-56bbd9b675-5xsxl" event={"ID":"3f3b16db-55fe-4505-9f9f-13bbfe486000","Type":"ContainerStarted","Data":"31d155fa6a70dbc1481d5f57008ef9ba6034ef1a50c22d33253fd32df44fbbeb"}
Oct 30 22:38:46.961410 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:46.961353    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-999470/test-model-999470-kserve-56bbd9b675-5xsxl" podStartSLOduration=1.9613368580000001 podStartE2EDuration="1.961336858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:38:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:38:46.957238954 +0000 UTC m=+1910.883165163" watchObservedRunningTime="2026-10-30 22:38:46.961336858 +0000 UTC m=+1910.887263065"
Oct 30 22:38:47.141878 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:47.141840    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-999470/test-model-999470-kserve-56bbd9b675-5xsxl"
Oct 30 22:38:48.823748 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:48.823718    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c13db0-749698df58-xzmc9"]
Oct 30 22:38:48.828260 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:48.828243    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c13db0-749698df58-xzmc9"
Oct 30 22:38:48.842762 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:48.842743    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-c13db0-serving-cert\""
Oct 30 22:38:48.858964 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:48.858936    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c13db0-749698df58-xzmc9"]
Oct 30 22:38:48.978936 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:48.978899    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25d06fe7-35c8-464e-9f50-1cf18a613fa1-openshift-service-ca\") pod \"maas-api-e2e-inf-b-c13db0-749698df58-xzmc9\" (UID: \"25d06fe7-35c8-464e-9f50-1cf18a613fa1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c13db0-749698df58-xzmc9"
Oct 30 22:38:48.979118 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:48.979035    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25d06fe7-35c8-464e-9f50-1cf18a613fa1-maas-api-tls\") pod \"maas-api-e2e-inf-b-c13db0-749698df58-xzmc9\" (UID: \"25d06fe7-35c8-464e-9f50-1cf18a613fa1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c13db0-749698df58-xzmc9"
Oct 30 22:38:48.979188 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:48.979174    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25d06fe7-35c8-464e-9f50-1cf18a613fa1-metrics-certs\") pod \"maas-api-e2e-inf-b-c13db0-749698df58-xzmc9\" (UID: \"25d06fe7-35c8-464e-9f50-1cf18a613fa1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c13db0-749698df58-xzmc9"
Oct 30 22:38:48.979260 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:48.979246    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6b5t\" (UniqueName: \"kubernetes.io/projected/25d06fe7-35c8-464e-9f50-1cf18a613fa1-kube-api-access-n6b5t\") pod \"maas-api-e2e-inf-b-c13db0-749698df58-xzmc9\" (UID: \"25d06fe7-35c8-464e-9f50-1cf18a613fa1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c13db0-749698df58-xzmc9"
Oct 30 22:38:49.081304 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:49.081213    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25d06fe7-35c8-464e-9f50-1cf18a613fa1-maas-api-tls\") pod \"maas-api-e2e-inf-b-c13db0-749698df58-xzmc9\" (UID: \"25d06fe7-35c8-464e-9f50-1cf18a613fa1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c13db0-749698df58-xzmc9"
Oct 30 22:38:49.081467 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:49.081391    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25d06fe7-35c8-464e-9f50-1cf18a613fa1-metrics-certs\") pod \"maas-api-e2e-inf-b-c13db0-749698df58-xzmc9\" (UID: \"25d06fe7-35c8-464e-9f50-1cf18a613fa1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c13db0-749698df58-xzmc9"
Oct 30 22:38:49.081532 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:49.081509    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6b5t\" (UniqueName: \"kubernetes.io/projected/25d06fe7-35c8-464e-9f50-1cf18a613fa1-kube-api-access-n6b5t\") pod \"maas-api-e2e-inf-b-c13db0-749698df58-xzmc9\" (UID: \"25d06fe7-35c8-464e-9f50-1cf18a613fa1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c13db0-749698df58-xzmc9"
Oct 30 22:38:49.081614 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:49.081586    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25d06fe7-35c8-464e-9f50-1cf18a613fa1-openshift-service-ca\") pod \"maas-api-e2e-inf-b-c13db0-749698df58-xzmc9\" (UID: \"25d06fe7-35c8-464e-9f50-1cf18a613fa1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c13db0-749698df58-xzmc9"
Oct 30 22:38:49.082594 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:49.082566    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25d06fe7-35c8-464e-9f50-1cf18a613fa1-openshift-service-ca\") pod \"maas-api-e2e-inf-b-c13db0-749698df58-xzmc9\" (UID: \"25d06fe7-35c8-464e-9f50-1cf18a613fa1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c13db0-749698df58-xzmc9"
Oct 30 22:38:49.085285 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:49.085262    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25d06fe7-35c8-464e-9f50-1cf18a613fa1-metrics-certs\") pod \"maas-api-e2e-inf-b-c13db0-749698df58-xzmc9\" (UID: \"25d06fe7-35c8-464e-9f50-1cf18a613fa1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c13db0-749698df58-xzmc9"
Oct 30 22:38:49.085567 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:49.085539    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25d06fe7-35c8-464e-9f50-1cf18a613fa1-maas-api-tls\") pod \"maas-api-e2e-inf-b-c13db0-749698df58-xzmc9\" (UID: \"25d06fe7-35c8-464e-9f50-1cf18a613fa1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c13db0-749698df58-xzmc9"
Oct 30 22:38:49.111095 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:49.101320    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6b5t\" (UniqueName: \"kubernetes.io/projected/25d06fe7-35c8-464e-9f50-1cf18a613fa1-kube-api-access-n6b5t\") pod \"maas-api-e2e-inf-b-c13db0-749698df58-xzmc9\" (UID: \"25d06fe7-35c8-464e-9f50-1cf18a613fa1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c13db0-749698df58-xzmc9"
Oct 30 22:38:49.143444 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:49.143406    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c13db0-749698df58-xzmc9"
Oct 30 22:38:49.449860 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:49.449833    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c13db0-749698df58-xzmc9"]
Oct 30 22:38:49.451083 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:38:49.451014    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25d06fe7_35c8_464e_9f50_1cf18a613fa1.slice/crio-931d4a3105cee7b42a522c3af67bc0ba623c16b60601a3d852818a94b8f9d1e1 WatchSource:0}: Error finding container 931d4a3105cee7b42a522c3af67bc0ba623c16b60601a3d852818a94b8f9d1e1: Status 404 returned error can't find the container with id 931d4a3105cee7b42a522c3af67bc0ba623c16b60601a3d852818a94b8f9d1e1
Oct 30 22:38:49.939816 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:49.939775    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c13db0-749698df58-xzmc9" event={"ID":"25d06fe7-35c8-464e-9f50-1cf18a613fa1","Type":"ContainerStarted","Data":"3af11adf035bd793840717919bab7b231df6b494c000884f8a83df685e3414a5"}
Oct 30 22:38:49.940268 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:49.939873    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c13db0-749698df58-xzmc9" event={"ID":"25d06fe7-35c8-464e-9f50-1cf18a613fa1","Type":"ContainerStarted","Data":"931d4a3105cee7b42a522c3af67bc0ba623c16b60601a3d852818a94b8f9d1e1"}
Oct 30 22:38:49.940268 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:49.939993    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c13db0-749698df58-xzmc9"
Oct 30 22:38:49.989792 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:49.989741    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c13db0-749698df58-xzmc9" podStartSLOduration=1.597868235 podStartE2EDuration="1.989727011s" totalImagesPullingTime="391.858776ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:38:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:38:49.985439188 +0000 UTC m=+1913.911365423" watchObservedRunningTime="2026-10-30 22:38:49.989727011 +0000 UTC m=+1913.915653217"
Oct 30 22:38:54.365461 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:54.365430    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6bz92"]
Oct 30 22:38:55.955087 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:55.955016    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c13db0-749698df58-xzmc9"
Oct 30 22:38:56.148070 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:56.141486    4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-999470/test-model-999470-kserve-56bbd9b675-5xsxl"
Oct 30 22:38:56.170139 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:56.170102    4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-999470/test-model-999470-kserve-56bbd9b675-5xsxl"
Oct 30 22:38:56.300294 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:38:56.300135    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 22:38:57.026301 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:38:57.026275    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-999470/test-model-999470-kserve-56bbd9b675-5xsxl"
Oct 30 22:39:26.269190 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:39:26.269149    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 22:39:36.856777 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:39:36.856744    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-867f12-8467f5bbc9-xfx5x"]
Oct 30 22:39:36.861112 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:39:36.861089    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-867f12-8467f5bbc9-xfx5x"
Oct 30 22:39:36.875797 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:39:36.875771    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-867f12-serving-cert\""
Oct 30 22:39:36.892714 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:39:36.892685    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-867f12-8467f5bbc9-xfx5x"]
Oct 30 22:39:36.942825 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:39:36.942801    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ff74f16-8919-4c9c-9b49-f333618cdf65-metrics-certs\") pod \"maas-api-e2e-inf-b-867f12-8467f5bbc9-xfx5x\" (UID: \"5ff74f16-8919-4c9c-9b49-f333618cdf65\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-867f12-8467f5bbc9-xfx5x"
Oct 30 22:39:36.942962 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:39:36.942846    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ff74f16-8919-4c9c-9b49-f333618cdf65-openshift-service-ca\") pod \"maas-api-e2e-inf-b-867f12-8467f5bbc9-xfx5x\" (UID: \"5ff74f16-8919-4c9c-9b49-f333618cdf65\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-867f12-8467f5bbc9-xfx5x"
Oct 30 22:39:36.943092 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:39:36.943028    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ff74f16-8919-4c9c-9b49-f333618cdf65-maas-api-tls\") pod \"maas-api-e2e-inf-b-867f12-8467f5bbc9-xfx5x\" (UID: \"5ff74f16-8919-4c9c-9b49-f333618cdf65\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-867f12-8467f5bbc9-xfx5x"
Oct 30 22:39:36.943248 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:39:36.943226    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cnn4\" (UniqueName: \"kubernetes.io/projected/5ff74f16-8919-4c9c-9b49-f333618cdf65-kube-api-access-4cnn4\") pod \"maas-api-e2e-inf-b-867f12-8467f5bbc9-xfx5x\" (UID: \"5ff74f16-8919-4c9c-9b49-f333618cdf65\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-867f12-8467f5bbc9-xfx5x"
Oct 30 22:39:37.044863 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:39:37.044821    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cnn4\" (UniqueName: \"kubernetes.io/projected/5ff74f16-8919-4c9c-9b49-f333618cdf65-kube-api-access-4cnn4\") pod \"maas-api-e2e-inf-b-867f12-8467f5bbc9-xfx5x\" (UID: \"5ff74f16-8919-4c9c-9b49-f333618cdf65\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-867f12-8467f5bbc9-xfx5x"
Oct 30 22:39:37.045055 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:39:37.044981    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ff74f16-8919-4c9c-9b49-f333618cdf65-metrics-certs\") pod \"maas-api-e2e-inf-b-867f12-8467f5bbc9-xfx5x\" (UID: \"5ff74f16-8919-4c9c-9b49-f333618cdf65\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-867f12-8467f5bbc9-xfx5x"
Oct 30 22:39:37.055697 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:39:37.046265    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ff74f16-8919-4c9c-9b49-f333618cdf65-openshift-service-ca\") pod \"maas-api-e2e-inf-b-867f12-8467f5bbc9-xfx5x\" (UID: \"5ff74f16-8919-4c9c-9b49-f333618cdf65\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-867f12-8467f5bbc9-xfx5x"
Oct 30 22:39:37.055697 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:39:37.047818    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ff74f16-8919-4c9c-9b49-f333618cdf65-openshift-service-ca\") pod \"maas-api-e2e-inf-b-867f12-8467f5bbc9-xfx5x\" (UID: \"5ff74f16-8919-4c9c-9b49-f333618cdf65\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-867f12-8467f5bbc9-xfx5x"
Oct 30 22:39:37.055697 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:39:37.048836    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ff74f16-8919-4c9c-9b49-f333618cdf65-maas-api-tls\") pod \"maas-api-e2e-inf-b-867f12-8467f5bbc9-xfx5x\" (UID: \"5ff74f16-8919-4c9c-9b49-f333618cdf65\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-867f12-8467f5bbc9-xfx5x"
Oct 30 22:39:37.058629 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:39:37.058597    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ff74f16-8919-4c9c-9b49-f333618cdf65-metrics-certs\") pod \"maas-api-e2e-inf-b-867f12-8467f5bbc9-xfx5x\" (UID: \"5ff74f16-8919-4c9c-9b49-f333618cdf65\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-867f12-8467f5bbc9-xfx5x"
Oct 30 22:39:37.058924 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:39:37.058748    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ff74f16-8919-4c9c-9b49-f333618cdf65-maas-api-tls\") pod \"maas-api-e2e-inf-b-867f12-8467f5bbc9-xfx5x\" (UID: \"5ff74f16-8919-4c9c-9b49-f333618cdf65\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-867f12-8467f5bbc9-xfx5x"
Oct 30 22:39:37.069750 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:39:37.069717    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cnn4\" (UniqueName: \"kubernetes.io/projected/5ff74f16-8919-4c9c-9b49-f333618cdf65-kube-api-access-4cnn4\") pod \"maas-api-e2e-inf-b-867f12-8467f5bbc9-xfx5x\" (UID: \"5ff74f16-8919-4c9c-9b49-f333618cdf65\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-867f12-8467f5bbc9-xfx5x"
Oct 30 22:39:37.176286 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:39:37.176257    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-867f12-8467f5bbc9-xfx5x"
Oct 30 22:39:37.480633 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:39:37.480597    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-867f12-8467f5bbc9-xfx5x"]
Oct 30 22:39:37.483935 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:39:37.483908    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ff74f16_8919_4c9c_9b49_f333618cdf65.slice/crio-04208f4db4541fb84c808f4cf1037a9cdb3a262c99381615ee7b05688e1a71a6 WatchSource:0}: Error finding container 04208f4db4541fb84c808f4cf1037a9cdb3a262c99381615ee7b05688e1a71a6: Status 404 returned error can't find the container with id 04208f4db4541fb84c808f4cf1037a9cdb3a262c99381615ee7b05688e1a71a6
Oct 30 22:39:38.321271 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:39:38.321237    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-867f12-8467f5bbc9-xfx5x" event={"ID":"5ff74f16-8919-4c9c-9b49-f333618cdf65","Type":"ContainerStarted","Data":"2c10f038af7e963e324f19a53f8467b3f17a0f4fd09011d92c95d07f25babe5f"}
Oct 30 22:39:38.321650 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:39:38.321298    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-867f12-8467f5bbc9-xfx5x"
Oct 30 22:39:38.321650 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:39:38.321320    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-867f12-8467f5bbc9-xfx5x" event={"ID":"5ff74f16-8919-4c9c-9b49-f333618cdf65","Type":"ContainerStarted","Data":"04208f4db4541fb84c808f4cf1037a9cdb3a262c99381615ee7b05688e1a71a6"}
Oct 30 22:39:38.393416 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:39:38.391719    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-867f12-8467f5bbc9-xfx5x" podStartSLOduration=1.787378074 podStartE2EDuration="2.39170055s" totalImagesPullingTime="604.322476ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:39:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:39:38.365777637 +0000 UTC m=+1962.291703843" watchObservedRunningTime="2026-10-30 22:39:38.39170055 +0000 UTC m=+1962.317626758"
Oct 30 22:39:44.335116 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:39:44.335075    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-867f12-8467f5bbc9-xfx5x"
Oct 30 22:39:53.952815 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:39:53.952780    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-f0bb45/test-model-f0bb45-kserve-74fd875fb5-msp5w"]
Oct 30 22:39:53.957771 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:39:53.957749    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f0bb45/test-model-f0bb45-kserve-74fd875fb5-msp5w"
Oct 30 22:39:53.980598 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:39:53.980564    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-f0bb45/test-model-f0bb45-kserve-74fd875fb5-msp5w"]
Oct 30 22:39:53.987091 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:39:53.986333    4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-f0bb45\"/\"openshift-service-ca.crt\""
Oct 30 22:39:53.987091 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:39:53.986553    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-f0bb45\"/\"test-model-f0bb45-kserve-self-signed-certs\""
Oct 30 22:39:53.987091 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:39:53.986618    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-f0bb45\"/\"default-dockercfg-qmpsp\""
Oct 30 22:39:53.987091 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:39:53.986672    4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-f0bb45\"/\"kube-root-ca.crt\""
Oct 30 22:39:54.032695 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:39:54.032658    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bedea217-88bb-4871-9f6f-0eb57c9b0654-tls-certs\") pod \"test-model-f0bb45-kserve-74fd875fb5-msp5w\" (UID: \"bedea217-88bb-4871-9f6f-0eb57c9b0654\") " pod="ai-tenant-e2e-inf-b-f0bb45/test-model-f0bb45-kserve-74fd875fb5-msp5w"
Oct 30 22:39:54.032842 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:39:54.032725    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kd5r9\" (UniqueName: \"kubernetes.io/projected/bedea217-88bb-4871-9f6f-0eb57c9b0654-kube-api-access-kd5r9\") pod \"test-model-f0bb45-kserve-74fd875fb5-msp5w\" (UID: \"bedea217-88bb-4871-9f6f-0eb57c9b0654\") " pod="ai-tenant-e2e-inf-b-f0bb45/test-model-f0bb45-kserve-74fd875fb5-msp5w"
Oct 30 22:39:54.032842 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:39:54.032808    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bedea217-88bb-4871-9f6f-0eb57c9b0654-home\") pod \"test-model-f0bb45-kserve-74fd875fb5-msp5w\" (UID: \"bedea217-88bb-4871-9f6f-0eb57c9b0654\") " pod="ai-tenant-e2e-inf-b-f0bb45/test-model-f0bb45-kserve-74fd875fb5-msp5w"
Oct 30 22:39:54.032922 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:39:54.032902    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bedea217-88bb-4871-9f6f-0eb57c9b0654-model-cache\") pod \"test-model-f0bb45-kserve-74fd875fb5-msp5w\" (UID: \"bedea217-88bb-4871-9f6f-0eb57c9b0654\") " pod="ai-tenant-e2e-inf-b-f0bb45/test-model-f0bb45-kserve-74fd875fb5-msp5w"
Oct 30 22:39:54.032971 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:39:54.032929    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bedea217-88bb-4871-9f6f-0eb57c9b0654-dshm\") pod \"test-model-f0bb45-kserve-74fd875fb5-msp5w\" (UID: \"bedea217-88bb-4871-9f6f-0eb57c9b0654\") " pod="ai-tenant-e2e-inf-b-f0bb45/test-model-f0bb45-kserve-74fd875fb5-msp5w"
Oct 30 22:39:54.033167 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:39:54.033143    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bedea217-88bb-4871-9f6f-0eb57c9b0654-tmp-dir\") pod \"test-model-f0bb45-kserve-74fd875fb5-msp5w\" (UID: \"bedea217-88bb-4871-9f6f-0eb57c9b0654\") " pod="ai-tenant-e2e-inf-b-f0bb45/test-model-f0bb45-kserve-74fd875fb5-msp5w"
Oct 30 22:39:54.134794 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:39:54.134752    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kd5r9\" (UniqueName: \"kubernetes.io/projected/bedea217-88bb-4871-9f6f-0eb57c9b0654-kube-api-access-kd5r9\") pod \"test-model-f0bb45-kserve-74fd875fb5-msp5w\" (UID: \"bedea217-88bb-4871-9f6f-0eb57c9b0654\") " pod="ai-tenant-e2e-inf-b-f0bb45/test-model-f0bb45-kserve-74fd875fb5-msp5w"
Oct 30 22:39:54.134968 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:39:54.134809    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bedea217-88bb-4871-9f6f-0eb57c9b0654-home\") pod \"test-model-f0bb45-kserve-74fd875fb5-msp5w\" (UID: \"bedea217-88bb-4871-9f6f-0eb57c9b0654\") " pod="ai-tenant-e2e-inf-b-f0bb45/test-model-f0bb45-kserve-74fd875fb5-msp5w"
Oct 30 22:39:54.134968 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:39:54.134892    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bedea217-88bb-4871-9f6f-0eb57c9b0654-model-cache\") pod \"test-model-f0bb45-kserve-74fd875fb5-msp5w\" (UID: \"bedea217-88bb-4871-9f6f-0eb57c9b0654\") " pod="ai-tenant-e2e-inf-b-f0bb45/test-model-f0bb45-kserve-74fd875fb5-msp5w"
Oct 30 22:39:54.135123 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:39:54.135095    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bedea217-88bb-4871-9f6f-0eb57c9b0654-dshm\") pod \"test-model-f0bb45-kserve-74fd875fb5-msp5w\" (UID: \"bedea217-88bb-4871-9f6f-0eb57c9b0654\") " pod="ai-tenant-e2e-inf-b-f0bb45/test-model-f0bb45-kserve-74fd875fb5-msp5w"
Oct 30 22:39:54.135312 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:39:54.135293    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bedea217-88bb-4871-9f6f-0eb57c9b0654-tmp-dir\") pod \"test-model-f0bb45-kserve-74fd875fb5-msp5w\" (UID: \"bedea217-88bb-4871-9f6f-0eb57c9b0654\") " pod="ai-tenant-e2e-inf-b-f0bb45/test-model-f0bb45-kserve-74fd875fb5-msp5w"
Oct 30 22:39:54.135383 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:39:54.135325    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bedea217-88bb-4871-9f6f-0eb57c9b0654-model-cache\") pod \"test-model-f0bb45-kserve-74fd875fb5-msp5w\" (UID: \"bedea217-88bb-4871-9f6f-0eb57c9b0654\") " pod="ai-tenant-e2e-inf-b-f0bb45/test-model-f0bb45-kserve-74fd875fb5-msp5w"
Oct 30 22:39:54.135434 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:39:54.135388    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bedea217-88bb-4871-9f6f-0eb57c9b0654-home\") pod \"test-model-f0bb45-kserve-74fd875fb5-msp5w\" (UID: \"bedea217-88bb-4871-9f6f-0eb57c9b0654\") " pod="ai-tenant-e2e-inf-b-f0bb45/test-model-f0bb45-kserve-74fd875fb5-msp5w"
Oct 30 22:39:54.135489 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:39:54.135471    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bedea217-88bb-4871-9f6f-0eb57c9b0654-tls-certs\") pod \"test-model-f0bb45-kserve-74fd875fb5-msp5w\" (UID: \"bedea217-88bb-4871-9f6f-0eb57c9b0654\") " pod="ai-tenant-e2e-inf-b-f0bb45/test-model-f0bb45-kserve-74fd875fb5-msp5w"
Oct 30 22:39:54.135637 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:39:54.135618    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bedea217-88bb-4871-9f6f-0eb57c9b0654-tmp-dir\") pod \"test-model-f0bb45-kserve-74fd875fb5-msp5w\" (UID: \"bedea217-88bb-4871-9f6f-0eb57c9b0654\") " pod="ai-tenant-e2e-inf-b-f0bb45/test-model-f0bb45-kserve-74fd875fb5-msp5w"
Oct 30 22:39:54.137566 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:39:54.137539    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bedea217-88bb-4871-9f6f-0eb57c9b0654-dshm\") pod \"test-model-f0bb45-kserve-74fd875fb5-msp5w\" (UID: \"bedea217-88bb-4871-9f6f-0eb57c9b0654\") " pod="ai-tenant-e2e-inf-b-f0bb45/test-model-f0bb45-kserve-74fd875fb5-msp5w"
Oct 30 22:39:54.137965 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:39:54.137937    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bedea217-88bb-4871-9f6f-0eb57c9b0654-tls-certs\") pod \"test-model-f0bb45-kserve-74fd875fb5-msp5w\" (UID: \"bedea217-88bb-4871-9f6f-0eb57c9b0654\") " pod="ai-tenant-e2e-inf-b-f0bb45/test-model-f0bb45-kserve-74fd875fb5-msp5w"
Oct 30 22:39:54.153687 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:39:54.153664    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kd5r9\" (UniqueName: \"kubernetes.io/projected/bedea217-88bb-4871-9f6f-0eb57c9b0654-kube-api-access-kd5r9\") pod \"test-model-f0bb45-kserve-74fd875fb5-msp5w\" (UID: \"bedea217-88bb-4871-9f6f-0eb57c9b0654\") " pod="ai-tenant-e2e-inf-b-f0bb45/test-model-f0bb45-kserve-74fd875fb5-msp5w"
Oct 30 22:39:54.272439 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:39:54.272348    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f0bb45/test-model-f0bb45-kserve-74fd875fb5-msp5w"
Oct 30 22:39:54.672467 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:39:54.672443    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-f0bb45/test-model-f0bb45-kserve-74fd875fb5-msp5w"]
Oct 30 22:39:54.675258 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:39:54.675226    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbedea217_88bb_4871_9f6f_0eb57c9b0654.slice/crio-c507462ed9a8ee6b2c15ab4d0412b2f8589b1649f060fb6df5d4860cae6fd7d0 WatchSource:0}: Error finding container c507462ed9a8ee6b2c15ab4d0412b2f8589b1649f060fb6df5d4860cae6fd7d0: Status 404 returned error can't find the container with id c507462ed9a8ee6b2c15ab4d0412b2f8589b1649f060fb6df5d4860cae6fd7d0
Oct 30 22:39:55.465371 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:39:55.465338    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f0bb45/test-model-f0bb45-kserve-74fd875fb5-msp5w" event={"ID":"bedea217-88bb-4871-9f6f-0eb57c9b0654","Type":"ContainerStarted","Data":"4a20f7fc0a4c844ec27cd91e46fb122cfde9bcb1d0836b208d688ff38d0bf715"}
Oct 30 22:39:55.465371 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:39:55.465383    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f0bb45/test-model-f0bb45-kserve-74fd875fb5-msp5w" event={"ID":"bedea217-88bb-4871-9f6f-0eb57c9b0654","Type":"ContainerStarted","Data":"c507462ed9a8ee6b2c15ab4d0412b2f8589b1649f060fb6df5d4860cae6fd7d0"}
Oct 30 22:39:55.508906 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:39:55.508847    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-f0bb45/test-model-f0bb45-kserve-74fd875fb5-msp5w" podStartSLOduration=2.508829394 podStartE2EDuration="2.508829394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:39:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:39:55.504521799 +0000 UTC m=+1979.430448021" watchObservedRunningTime="2026-10-30 22:39:55.508829394 +0000 UTC m=+1979.434755603"
Oct 30 22:39:56.267402 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:39:56.267366    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 22:39:56.273798 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:39:56.273769    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-f0bb45/test-model-f0bb45-kserve-74fd875fb5-msp5w"
Oct 30 22:39:58.981485 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:39:58.981451    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-3acfdb-76fffb9796-mqnhp"]
Oct 30 22:39:58.986197 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:39:58.986176    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3acfdb-76fffb9796-mqnhp"
Oct 30 22:39:59.015554 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:39:59.015529    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-3acfdb-serving-cert\""
Oct 30 22:39:59.021418 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:39:59.021381    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-3acfdb-76fffb9796-mqnhp"]
Oct 30 22:39:59.113109 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:39:59.113085    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ede416f-63b4-4f4b-802f-22f1f3183e9f-maas-api-tls\") pod \"maas-api-e2e-inf-b-3acfdb-76fffb9796-mqnhp\" (UID: \"0ede416f-63b4-4f4b-802f-22f1f3183e9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3acfdb-76fffb9796-mqnhp"
Oct 30 22:39:59.113250 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:39:59.113148    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4zbj\" (UniqueName: \"kubernetes.io/projected/0ede416f-63b4-4f4b-802f-22f1f3183e9f-kube-api-access-x4zbj\") pod \"maas-api-e2e-inf-b-3acfdb-76fffb9796-mqnhp\" (UID: \"0ede416f-63b4-4f4b-802f-22f1f3183e9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3acfdb-76fffb9796-mqnhp"
Oct 30 22:39:59.113300 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:39:59.113272    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ede416f-63b4-4f4b-802f-22f1f3183e9f-openshift-service-ca\") pod \"maas-api-e2e-inf-b-3acfdb-76fffb9796-mqnhp\" (UID: \"0ede416f-63b4-4f4b-802f-22f1f3183e9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3acfdb-76fffb9796-mqnhp"
Oct 30 22:39:59.113353 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:39:59.113325    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ede416f-63b4-4f4b-802f-22f1f3183e9f-metrics-certs\") pod \"maas-api-e2e-inf-b-3acfdb-76fffb9796-mqnhp\" (UID: \"0ede416f-63b4-4f4b-802f-22f1f3183e9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3acfdb-76fffb9796-mqnhp"
Oct 30 22:39:59.214758 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:39:59.214707    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ede416f-63b4-4f4b-802f-22f1f3183e9f-metrics-certs\") pod \"maas-api-e2e-inf-b-3acfdb-76fffb9796-mqnhp\" (UID: \"0ede416f-63b4-4f4b-802f-22f1f3183e9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3acfdb-76fffb9796-mqnhp"
Oct 30 22:39:59.214937 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:39:59.214922    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ede416f-63b4-4f4b-802f-22f1f3183e9f-maas-api-tls\") pod \"maas-api-e2e-inf-b-3acfdb-76fffb9796-mqnhp\" (UID: \"0ede416f-63b4-4f4b-802f-22f1f3183e9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3acfdb-76fffb9796-mqnhp"
Oct 30 22:39:59.215120 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:39:59.215093    4477 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-3acfdb-serving-cert: secret "maas-api-e2e-inf-b-3acfdb-serving-cert" not found
Oct 30 22:39:59.215354 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:39:59.215121    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4zbj\" (UniqueName: \"kubernetes.io/projected/0ede416f-63b4-4f4b-802f-22f1f3183e9f-kube-api-access-x4zbj\") pod \"maas-api-e2e-inf-b-3acfdb-76fffb9796-mqnhp\" (UID: \"0ede416f-63b4-4f4b-802f-22f1f3183e9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3acfdb-76fffb9796-mqnhp"
Oct 30 22:39:59.215473 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:39:59.215458    4477 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ede416f-63b4-4f4b-802f-22f1f3183e9f-maas-api-tls podName:0ede416f-63b4-4f4b-802f-22f1f3183e9f nodeName:}" failed. No retries permitted until 2026-10-30 22:39:59.715326731 +0000 UTC m=+1983.641252920 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0ede416f-63b4-4f4b-802f-22f1f3183e9f-maas-api-tls") pod "maas-api-e2e-inf-b-3acfdb-76fffb9796-mqnhp" (UID: "0ede416f-63b4-4f4b-802f-22f1f3183e9f") : secret "maas-api-e2e-inf-b-3acfdb-serving-cert" not found
Oct 30 22:39:59.215707 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:39:59.215688    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ede416f-63b4-4f4b-802f-22f1f3183e9f-openshift-service-ca\") pod \"maas-api-e2e-inf-b-3acfdb-76fffb9796-mqnhp\" (UID: \"0ede416f-63b4-4f4b-802f-22f1f3183e9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3acfdb-76fffb9796-mqnhp"
Oct 30 22:39:59.216778 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:39:59.216724    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ede416f-63b4-4f4b-802f-22f1f3183e9f-openshift-service-ca\") pod \"maas-api-e2e-inf-b-3acfdb-76fffb9796-mqnhp\" (UID: \"0ede416f-63b4-4f4b-802f-22f1f3183e9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3acfdb-76fffb9796-mqnhp"
Oct 30 22:39:59.218401 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:39:59.218377    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ede416f-63b4-4f4b-802f-22f1f3183e9f-metrics-certs\") pod \"maas-api-e2e-inf-b-3acfdb-76fffb9796-mqnhp\" (UID: \"0ede416f-63b4-4f4b-802f-22f1f3183e9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3acfdb-76fffb9796-mqnhp"
Oct 30 22:39:59.239447 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:39:59.239375    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4zbj\" (UniqueName: \"kubernetes.io/projected/0ede416f-63b4-4f4b-802f-22f1f3183e9f-kube-api-access-x4zbj\") pod \"maas-api-e2e-inf-b-3acfdb-76fffb9796-mqnhp\" (UID: \"0ede416f-63b4-4f4b-802f-22f1f3183e9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3acfdb-76fffb9796-mqnhp"
Oct 30 22:39:59.722141 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:39:59.722112    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ede416f-63b4-4f4b-802f-22f1f3183e9f-maas-api-tls\") pod \"maas-api-e2e-inf-b-3acfdb-76fffb9796-mqnhp\" (UID: \"0ede416f-63b4-4f4b-802f-22f1f3183e9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3acfdb-76fffb9796-mqnhp"
Oct 30 22:39:59.724598 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:39:59.724571    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ede416f-63b4-4f4b-802f-22f1f3183e9f-maas-api-tls\") pod \"maas-api-e2e-inf-b-3acfdb-76fffb9796-mqnhp\" (UID: \"0ede416f-63b4-4f4b-802f-22f1f3183e9f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3acfdb-76fffb9796-mqnhp"
Oct 30 22:39:59.900202 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:39:59.900164    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3acfdb-76fffb9796-mqnhp"
Oct 30 22:40:00.311183 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:40:00.310694    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-3acfdb-76fffb9796-mqnhp"]
Oct 30 22:40:00.513803 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:40:00.513761    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3acfdb-76fffb9796-mqnhp" event={"ID":"0ede416f-63b4-4f4b-802f-22f1f3183e9f","Type":"ContainerStarted","Data":"79f2c3d7b301a896b9b12af3a632328aaebfade6e602db5a5ba4546f700b85e5"}
Oct 30 22:40:01.525120 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:40:01.525076    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3acfdb-76fffb9796-mqnhp" event={"ID":"0ede416f-63b4-4f4b-802f-22f1f3183e9f","Type":"ContainerStarted","Data":"8f17694c7b720a70b3bf5138aadb0e35d8fbfe90109c7873dbb8f1e36fa2ff8e"}
Oct 30 22:40:01.525619 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:40:01.525325    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3acfdb-76fffb9796-mqnhp"
Oct 30 22:40:01.565583 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:40:01.565537    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3acfdb-76fffb9796-mqnhp" podStartSLOduration=3.236836755 podStartE2EDuration="3.565524971s" totalImagesPullingTime="328.688216ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:39:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:40:01.564210223 +0000 UTC m=+1985.490136430" watchObservedRunningTime="2026-10-30 22:40:01.565524971 +0000 UTC m=+1985.491451197"
Oct 30 22:40:04.145680 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:40:04.144986    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6bz92"]
Oct 30 22:40:04.273280 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:40:04.273236    4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-f0bb45/test-model-f0bb45-kserve-74fd875fb5-msp5w"
Oct 30 22:40:04.291491 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:40:04.291461    4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-f0bb45/test-model-f0bb45-kserve-74fd875fb5-msp5w"
Oct 30 22:40:04.564628 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:40:04.564546    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-f0bb45/test-model-f0bb45-kserve-74fd875fb5-msp5w"
Oct 30 22:40:07.537826 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:40:07.537796    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3acfdb-76fffb9796-mqnhp"
Oct 30 22:40:26.267706 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:40:26.267668    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 22:40:55.035450 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:40:55.035370    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-vtwp6"]
Oct 30 22:40:55.035993 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:40:55.035577    4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-vtwp6" podUID="623d861e-ea33-4cd5-af9e-3b29b937e335" containerName="istio-proxy" containerID="cri-o://be0bd308f5ff525861432f5dc7749174652c7a6e2bfaa27dda66b70eeffd202a" gracePeriod=30
Oct 30 22:40:56.275836 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:40:56.275798    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 22:41:00.329837 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:41:00.329813    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-vtwp6"
Oct 30 22:41:00.420288 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:41:00.420203    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/623d861e-ea33-4cd5-af9e-3b29b937e335-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/623d861e-ea33-4cd5-af9e-3b29b937e335-credential-socket\") pod \"623d861e-ea33-4cd5-af9e-3b29b937e335\" (UID: \"623d861e-ea33-4cd5-af9e-3b29b937e335\") "
Oct 30 22:41:00.420462 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:41:00.420292    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/623d861e-ea33-4cd5-af9e-3b29b937e335-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/623d861e-ea33-4cd5-af9e-3b29b937e335-istio-data\") pod \"623d861e-ea33-4cd5-af9e-3b29b937e335\" (UID: \"623d861e-ea33-4cd5-af9e-3b29b937e335\") "
Oct 30 22:41:00.420462 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:41:00.420400    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/623d861e-ea33-4cd5-af9e-3b29b937e335-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/623d861e-ea33-4cd5-af9e-3b29b937e335-workload-socket\") pod \"623d861e-ea33-4cd5-af9e-3b29b937e335\" (UID: \"623d861e-ea33-4cd5-af9e-3b29b937e335\") "
Oct 30 22:41:00.420462 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:41:00.420453    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/623d861e-ea33-4cd5-af9e-3b29b937e335-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/623d861e-ea33-4cd5-af9e-3b29b937e335-istiod-ca-cert\") pod \"623d861e-ea33-4cd5-af9e-3b29b937e335\" (UID: \"623d861e-ea33-4cd5-af9e-3b29b937e335\") "
Oct 30 22:41:00.420645 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:41:00.420530    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/623d861e-ea33-4cd5-af9e-3b29b937e335-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/623d861e-ea33-4cd5-af9e-3b29b937e335-workload-certs\") pod \"623d861e-ea33-4cd5-af9e-3b29b937e335\" (UID: \"623d861e-ea33-4cd5-af9e-3b29b937e335\") "
Oct 30 22:41:00.420645 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:41:00.420572    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/623d861e-ea33-4cd5-af9e-3b29b937e335-istio-token\" (UniqueName: \"kubernetes.io/projected/623d861e-ea33-4cd5-af9e-3b29b937e335-istio-token\") pod \"623d861e-ea33-4cd5-af9e-3b29b937e335\" (UID: \"623d861e-ea33-4cd5-af9e-3b29b937e335\") "
Oct 30 22:41:00.420753 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:41:00.420649    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/623d861e-ea33-4cd5-af9e-3b29b937e335-kube-api-access-hd7cj\" (UniqueName: \"kubernetes.io/projected/623d861e-ea33-4cd5-af9e-3b29b937e335-kube-api-access-hd7cj\") pod \"623d861e-ea33-4cd5-af9e-3b29b937e335\" (UID: \"623d861e-ea33-4cd5-af9e-3b29b937e335\") "
Oct 30 22:41:00.420753 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:41:00.420685    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/623d861e-ea33-4cd5-af9e-3b29b937e335-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/623d861e-ea33-4cd5-af9e-3b29b937e335-istio-podinfo\") pod \"623d861e-ea33-4cd5-af9e-3b29b937e335\" (UID: \"623d861e-ea33-4cd5-af9e-3b29b937e335\") "
Oct 30 22:41:00.420856 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:41:00.420763    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/623d861e-ea33-4cd5-af9e-3b29b937e335-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/623d861e-ea33-4cd5-af9e-3b29b937e335-istio-envoy\") pod \"623d861e-ea33-4cd5-af9e-3b29b937e335\" (UID: \"623d861e-ea33-4cd5-af9e-3b29b937e335\") "
Oct 30 22:41:00.422035 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:41:00.422010    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/623d861e-ea33-4cd5-af9e-3b29b937e335-istio-data" pod "623d861e-ea33-4cd5-af9e-3b29b937e335" (UID: "623d861e-ea33-4cd5-af9e-3b29b937e335"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:41:00.422247 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:41:00.422024    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/623d861e-ea33-4cd5-af9e-3b29b937e335-istiod-ca-cert" pod "623d861e-ea33-4cd5-af9e-3b29b937e335" (UID: "623d861e-ea33-4cd5-af9e-3b29b937e335"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:41:00.422407 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:41:00.422232    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/623d861e-ea33-4cd5-af9e-3b29b937e335-workload-socket" pod "623d861e-ea33-4cd5-af9e-3b29b937e335" (UID: "623d861e-ea33-4cd5-af9e-3b29b937e335"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:41:00.422407 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:41:00.422248    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/623d861e-ea33-4cd5-af9e-3b29b937e335-workload-certs" pod "623d861e-ea33-4cd5-af9e-3b29b937e335" (UID: "623d861e-ea33-4cd5-af9e-3b29b937e335"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:41:00.422939 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:41:00.422886    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/623d861e-ea33-4cd5-af9e-3b29b937e335-credential-socket" pod "623d861e-ea33-4cd5-af9e-3b29b937e335" (UID: "623d861e-ea33-4cd5-af9e-3b29b937e335"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:41:00.424911 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:41:00.424881    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/623d861e-ea33-4cd5-af9e-3b29b937e335-istio-envoy" pod "623d861e-ea33-4cd5-af9e-3b29b937e335" (UID: "623d861e-ea33-4cd5-af9e-3b29b937e335"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:41:00.424911 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:41:00.424902    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/623d861e-ea33-4cd5-af9e-3b29b937e335-istio-token" pod "623d861e-ea33-4cd5-af9e-3b29b937e335" (UID: "623d861e-ea33-4cd5-af9e-3b29b937e335"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:41:00.425105 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:41:00.424965    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/623d861e-ea33-4cd5-af9e-3b29b937e335-istio-podinfo" pod "623d861e-ea33-4cd5-af9e-3b29b937e335" (UID: "623d861e-ea33-4cd5-af9e-3b29b937e335"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:41:00.425483 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:41:00.425458    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/623d861e-ea33-4cd5-af9e-3b29b937e335-kube-api-access-hd7cj" pod "623d861e-ea33-4cd5-af9e-3b29b937e335" (UID: "623d861e-ea33-4cd5-af9e-3b29b937e335"). InnerVolumeSpecName "kube-api-access-hd7cj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:41:00.523174 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:41:00.523141    4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hd7cj\" (UniqueName: \"kubernetes.io/projected/623d861e-ea33-4cd5-af9e-3b29b937e335-kube-api-access-hd7cj\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:41:00.523174 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:41:00.523172    4477 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/623d861e-ea33-4cd5-af9e-3b29b937e335-istio-podinfo\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:41:00.523174 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:41:00.523181    4477 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/623d861e-ea33-4cd5-af9e-3b29b937e335-istio-envoy\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:41:00.523419 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:41:00.523189    4477 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/623d861e-ea33-4cd5-af9e-3b29b937e335-credential-socket\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:41:00.523419 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:41:00.523198    4477 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/623d861e-ea33-4cd5-af9e-3b29b937e335-istio-data\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:41:00.523419 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:41:00.523206    4477 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/623d861e-ea33-4cd5-af9e-3b29b937e335-workload-socket\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:41:00.523419 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:41:00.523215    4477 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/623d861e-ea33-4cd5-af9e-3b29b937e335-istiod-ca-cert\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:41:00.523419 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:41:00.523223    4477 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/623d861e-ea33-4cd5-af9e-3b29b937e335-workload-certs\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:41:00.523419 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:41:00.523232    4477 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/623d861e-ea33-4cd5-af9e-3b29b937e335-istio-token\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:41:00.980945 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:41:00.980909    4477 generic.go:356] "Generic (PLEG): container finished" podID="623d861e-ea33-4cd5-af9e-3b29b937e335" containerID="be0bd308f5ff525861432f5dc7749174652c7a6e2bfaa27dda66b70eeffd202a" exitCode=0
Oct 30 22:41:00.981174 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:41:00.980976    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-vtwp6"
Oct 30 22:41:00.981174 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:41:00.980986    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-vtwp6" event={"ID":"623d861e-ea33-4cd5-af9e-3b29b937e335","Type":"ContainerDied","Data":"be0bd308f5ff525861432f5dc7749174652c7a6e2bfaa27dda66b70eeffd202a"}
Oct 30 22:41:00.981174 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:41:00.981093    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-vtwp6" event={"ID":"623d861e-ea33-4cd5-af9e-3b29b937e335","Type":"ContainerDied","Data":"962ead7ef6210c04a01ce515730436cadc44075e5ecd42658435224cddfb6862"}
Oct 30 22:41:00.981174 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:41:00.981119    4477 scope.go:122] "RemoveContainer" containerID="be0bd308f5ff525861432f5dc7749174652c7a6e2bfaa27dda66b70eeffd202a"
Oct 30 22:41:00.991933 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:41:00.991917    4477 scope.go:122] "RemoveContainer" containerID="be0bd308f5ff525861432f5dc7749174652c7a6e2bfaa27dda66b70eeffd202a"
Oct 30 22:41:00.992214 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:41:00.992196    4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be0bd308f5ff525861432f5dc7749174652c7a6e2bfaa27dda66b70eeffd202a\": container with ID starting with be0bd308f5ff525861432f5dc7749174652c7a6e2bfaa27dda66b70eeffd202a not found: ID does not exist" containerID="be0bd308f5ff525861432f5dc7749174652c7a6e2bfaa27dda66b70eeffd202a"
Oct 30 22:41:00.992283 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:41:00.992221    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be0bd308f5ff525861432f5dc7749174652c7a6e2bfaa27dda66b70eeffd202a"} err="failed to get container status \"be0bd308f5ff525861432f5dc7749174652c7a6e2bfaa27dda66b70eeffd202a\": rpc error: code = NotFound desc = could not find container \"be0bd308f5ff525861432f5dc7749174652c7a6e2bfaa27dda66b70eeffd202a\": container with ID starting with be0bd308f5ff525861432f5dc7749174652c7a6e2bfaa27dda66b70eeffd202a not found: ID does not exist"
Oct 30 22:41:01.050011 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:41:01.049981    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-vtwp6"]
Oct 30 22:41:01.053581 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:41:01.053553    4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-vtwp6"]
Oct 30 22:41:02.196096 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:41:02.196064    4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="623d861e-ea33-4cd5-af9e-3b29b937e335" path="/var/lib/kubelet/pods/623d861e-ea33-4cd5-af9e-3b29b937e335/volumes"
Oct 30 22:41:10.026600 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:41:10.023793    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6bz92"]
Oct 30 22:41:13.466586 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:41:13.466555    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-867f12/test-model-867f12-kserve-6694d95cf9-jcxgf"]
Oct 30 22:41:13.467638 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:41:13.467619    4477 cpu_manager.go:414] "containerMap: removing container" podUID="623d861e-ea33-4cd5-af9e-3b29b937e335" containerName="istio-proxy"
Oct 30 22:41:13.467694 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:41:13.467642    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="623d861e-ea33-4cd5-af9e-3b29b937e335" containerName="istio-proxy"
Oct 30 22:41:13.467879 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:41:13.467867    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="623d861e-ea33-4cd5-af9e-3b29b937e335" containerName="istio-proxy"
Oct 30 22:41:13.474220 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:41:13.474197    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-867f12/test-model-867f12-kserve-6694d95cf9-jcxgf"
Oct 30 22:41:13.493475 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:41:13.492493    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-867f12/test-model-867f12-kserve-6694d95cf9-jcxgf"]
Oct 30 22:41:13.500412 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:41:13.500387    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-867f12\"/\"default-dockercfg-gnbff\""
Oct 30 22:41:13.501034 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:41:13.500954    4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-867f12\"/\"kube-root-ca.crt\""
Oct 30 22:41:13.501034 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:41:13.500966    4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-867f12\"/\"openshift-service-ca.crt\""
Oct 30 22:41:13.501204 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:41:13.501092    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-867f12\"/\"test-model-867f12-kserve-self-signed-certs\""
Oct 30 22:41:13.629373 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:41:13.629338    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f6746af-0eb9-4a46-a201-1e17626a9f81-model-cache\") pod \"test-model-867f12-kserve-6694d95cf9-jcxgf\" (UID: \"5f6746af-0eb9-4a46-a201-1e17626a9f81\") " pod="ai-tenant-e2e-inf-b-867f12/test-model-867f12-kserve-6694d95cf9-jcxgf"
Oct 30 22:41:13.629531 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:41:13.629382    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f6746af-0eb9-4a46-a201-1e17626a9f81-home\") pod \"test-model-867f12-kserve-6694d95cf9-jcxgf\" (UID: \"5f6746af-0eb9-4a46-a201-1e17626a9f81\") " pod="ai-tenant-e2e-inf-b-867f12/test-model-867f12-kserve-6694d95cf9-jcxgf"
Oct 30 22:41:13.629531 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:41:13.629488    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5bch\" (UniqueName: \"kubernetes.io/projected/5f6746af-0eb9-4a46-a201-1e17626a9f81-kube-api-access-w5bch\") pod \"test-model-867f12-kserve-6694d95cf9-jcxgf\" (UID: \"5f6746af-0eb9-4a46-a201-1e17626a9f81\") " pod="ai-tenant-e2e-inf-b-867f12/test-model-867f12-kserve-6694d95cf9-jcxgf"
Oct 30 22:41:13.629641 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:41:13.629616    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f6746af-0eb9-4a46-a201-1e17626a9f81-dshm\") pod \"test-model-867f12-kserve-6694d95cf9-jcxgf\" (UID: \"5f6746af-0eb9-4a46-a201-1e17626a9f81\") " pod="ai-tenant-e2e-inf-b-867f12/test-model-867f12-kserve-6694d95cf9-jcxgf"
Oct 30 22:41:13.629814 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:41:13.629794    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f6746af-0eb9-4a46-a201-1e17626a9f81-tmp-dir\") pod \"test-model-867f12-kserve-6694d95cf9-jcxgf\" (UID: \"5f6746af-0eb9-4a46-a201-1e17626a9f81\") " pod="ai-tenant-e2e-inf-b-867f12/test-model-867f12-kserve-6694d95cf9-jcxgf"
Oct 30 22:41:13.629892 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:41:13.629879    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f6746af-0eb9-4a46-a201-1e17626a9f81-tls-certs\") pod \"test-model-867f12-kserve-6694d95cf9-jcxgf\" (UID: \"5f6746af-0eb9-4a46-a201-1e17626a9f81\") " pod="ai-tenant-e2e-inf-b-867f12/test-model-867f12-kserve-6694d95cf9-jcxgf"
Oct 30 22:41:13.736107 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:41:13.731339    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f6746af-0eb9-4a46-a201-1e17626a9f81-tmp-dir\") pod \"test-model-867f12-kserve-6694d95cf9-jcxgf\" (UID: \"5f6746af-0eb9-4a46-a201-1e17626a9f81\") " pod="ai-tenant-e2e-inf-b-867f12/test-model-867f12-kserve-6694d95cf9-jcxgf"
Oct 30 22:41:13.736107 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:41:13.731435    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f6746af-0eb9-4a46-a201-1e17626a9f81-tls-certs\") pod \"test-model-867f12-kserve-6694d95cf9-jcxgf\" (UID: \"5f6746af-0eb9-4a46-a201-1e17626a9f81\") " pod="ai-tenant-e2e-inf-b-867f12/test-model-867f12-kserve-6694d95cf9-jcxgf"
Oct 30 22:41:13.736107 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:41:13.731595    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f6746af-0eb9-4a46-a201-1e17626a9f81-model-cache\") pod \"test-model-867f12-kserve-6694d95cf9-jcxgf\" (UID: \"5f6746af-0eb9-4a46-a201-1e17626a9f81\") " pod="ai-tenant-e2e-inf-b-867f12/test-model-867f12-kserve-6694d95cf9-jcxgf"
Oct 30 22:41:13.736107 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:41:13.731637    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f6746af-0eb9-4a46-a201-1e17626a9f81-home\") pod \"test-model-867f12-kserve-6694d95cf9-jcxgf\" (UID: \"5f6746af-0eb9-4a46-a201-1e17626a9f81\") " pod="ai-tenant-e2e-inf-b-867f12/test-model-867f12-kserve-6694d95cf9-jcxgf"
Oct 30 22:41:13.736107 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:41:13.731720    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5bch\" (UniqueName: \"kubernetes.io/projected/5f6746af-0eb9-4a46-a201-1e17626a9f81-kube-api-access-w5bch\") pod \"test-model-867f12-kserve-6694d95cf9-jcxgf\" (UID: \"5f6746af-0eb9-4a46-a201-1e17626a9f81\") " pod="ai-tenant-e2e-inf-b-867f12/test-model-867f12-kserve-6694d95cf9-jcxgf"
Oct 30 22:41:13.736107 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:41:13.731755    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f6746af-0eb9-4a46-a201-1e17626a9f81-dshm\") pod \"test-model-867f12-kserve-6694d95cf9-jcxgf\" (UID: \"5f6746af-0eb9-4a46-a201-1e17626a9f81\") " pod="ai-tenant-e2e-inf-b-867f12/test-model-867f12-kserve-6694d95cf9-jcxgf"
Oct 30 22:41:13.736107 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:41:13.732659    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f6746af-0eb9-4a46-a201-1e17626a9f81-model-cache\") pod \"test-model-867f12-kserve-6694d95cf9-jcxgf\" (UID: \"5f6746af-0eb9-4a46-a201-1e17626a9f81\") " pod="ai-tenant-e2e-inf-b-867f12/test-model-867f12-kserve-6694d95cf9-jcxgf"
Oct 30 22:41:13.736107 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:41:13.732919    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f6746af-0eb9-4a46-a201-1e17626a9f81-tmp-dir\") pod \"test-model-867f12-kserve-6694d95cf9-jcxgf\" (UID: \"5f6746af-0eb9-4a46-a201-1e17626a9f81\") " pod="ai-tenant-e2e-inf-b-867f12/test-model-867f12-kserve-6694d95cf9-jcxgf"
Oct 30 22:41:13.736107 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:41:13.733536    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f6746af-0eb9-4a46-a201-1e17626a9f81-home\") pod \"test-model-867f12-kserve-6694d95cf9-jcxgf\" (UID: \"5f6746af-0eb9-4a46-a201-1e17626a9f81\") " pod="ai-tenant-e2e-inf-b-867f12/test-model-867f12-kserve-6694d95cf9-jcxgf"
Oct 30 22:41:13.739338 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:41:13.739311    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f6746af-0eb9-4a46-a201-1e17626a9f81-tls-certs\") pod \"test-model-867f12-kserve-6694d95cf9-jcxgf\" (UID: \"5f6746af-0eb9-4a46-a201-1e17626a9f81\") " pod="ai-tenant-e2e-inf-b-867f12/test-model-867f12-kserve-6694d95cf9-jcxgf"
Oct 30 22:41:13.764831 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:41:13.764797    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5bch\" (UniqueName: \"kubernetes.io/projected/5f6746af-0eb9-4a46-a201-1e17626a9f81-kube-api-access-w5bch\") pod \"test-model-867f12-kserve-6694d95cf9-jcxgf\" (UID: \"5f6746af-0eb9-4a46-a201-1e17626a9f81\") " pod="ai-tenant-e2e-inf-b-867f12/test-model-867f12-kserve-6694d95cf9-jcxgf"
Oct 30 22:41:13.769651 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:41:13.769621    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f6746af-0eb9-4a46-a201-1e17626a9f81-dshm\") pod \"test-model-867f12-kserve-6694d95cf9-jcxgf\" (UID: \"5f6746af-0eb9-4a46-a201-1e17626a9f81\") " pod="ai-tenant-e2e-inf-b-867f12/test-model-867f12-kserve-6694d95cf9-jcxgf"
Oct 30 22:41:13.807643 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:41:13.807610    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-867f12/test-model-867f12-kserve-6694d95cf9-jcxgf"
Oct 30 22:41:14.140572 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:41:14.140538    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-867f12/test-model-867f12-kserve-6694d95cf9-jcxgf"]
Oct 30 22:41:14.142076 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:41:14.142032    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5f6746af_0eb9_4a46_a201_1e17626a9f81.slice/crio-e5a72d53a33090efd1e94dbf67f02a664dbc5fcbb6f704c42328f3f86f029f03 WatchSource:0}: Error finding container e5a72d53a33090efd1e94dbf67f02a664dbc5fcbb6f704c42328f3f86f029f03: Status 404 returned error can't find the container with id e5a72d53a33090efd1e94dbf67f02a664dbc5fcbb6f704c42328f3f86f029f03
Oct 30 22:41:15.083736 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:41:15.083689    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-867f12/test-model-867f12-kserve-6694d95cf9-jcxgf" event={"ID":"5f6746af-0eb9-4a46-a201-1e17626a9f81","Type":"ContainerStarted","Data":"31211b50d977f546c6972dfed22b86eae487567be864c6d9c531d3ad35a50582"}
Oct 30 22:41:15.084202 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:41:15.083744    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-867f12/test-model-867f12-kserve-6694d95cf9-jcxgf" event={"ID":"5f6746af-0eb9-4a46-a201-1e17626a9f81","Type":"ContainerStarted","Data":"e5a72d53a33090efd1e94dbf67f02a664dbc5fcbb6f704c42328f3f86f029f03"}
Oct 30 22:41:15.125893 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:41:15.125838    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-867f12/test-model-867f12-kserve-6694d95cf9-jcxgf" podStartSLOduration=2.125824233 podStartE2EDuration="2.125824233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:41:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:41:15.121990069 +0000 UTC m=+2059.047916276" watchObservedRunningTime="2026-10-30 22:41:15.125824233 +0000 UTC m=+2059.051750439"
Oct 30 22:41:15.808609 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:41:15.808576    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-867f12/test-model-867f12-kserve-6694d95cf9-jcxgf"
Oct 30 22:41:23.808497 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:41:23.808465    4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-867f12/test-model-867f12-kserve-6694d95cf9-jcxgf"
Oct 30 22:41:23.822157 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:41:23.822130    4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-867f12/test-model-867f12-kserve-6694d95cf9-jcxgf"
Oct 30 22:41:24.166312 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:41:24.166281    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-867f12/test-model-867f12-kserve-6694d95cf9-jcxgf"
Oct 30 22:41:26.267699 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:41:26.267664    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 22:41:39.747486 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:41:39.747456    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-3acfdb/test-model-3acfdb-kserve-79856cb545-b29mk"]
Oct 30 22:41:39.751900 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:41:39.751874    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-3acfdb/test-model-3acfdb-kserve-79856cb545-b29mk"
Oct 30 22:41:39.775298 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:41:39.775271    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-3acfdb/test-model-3acfdb-kserve-79856cb545-b29mk"]
Oct 30 22:41:39.781355 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:41:39.781320    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-3acfdb\"/\"default-dockercfg-l8qhf\""
Oct 30 22:41:39.781539 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:41:39.781364    4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-3acfdb\"/\"openshift-service-ca.crt\""
Oct 30 22:41:39.781539 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:41:39.781391    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-3acfdb\"/\"test-model-3acfdb-kserve-self-signed-certs\""
Oct 30 22:41:39.781726 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:41:39.781435    4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-3acfdb\"/\"kube-root-ca.crt\""
Oct 30 22:41:39.802830 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:41:39.802803    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13dfee80-6a6d-4b97-be9d-728a85a59e5b-tls-certs\") pod \"test-model-3acfdb-kserve-79856cb545-b29mk\" (UID: \"13dfee80-6a6d-4b97-be9d-728a85a59e5b\") " pod="ai-tenant-e2e-inf-b-3acfdb/test-model-3acfdb-kserve-79856cb545-b29mk"
Oct 30 22:41:39.803080 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:41:39.803020    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13dfee80-6a6d-4b97-be9d-728a85a59e5b-dshm\") pod \"test-model-3acfdb-kserve-79856cb545-b29mk\" (UID: \"13dfee80-6a6d-4b97-be9d-728a85a59e5b\") " pod="ai-tenant-e2e-inf-b-3acfdb/test-model-3acfdb-kserve-79856cb545-b29mk"
Oct 30 22:41:39.803189 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:41:39.803159    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13dfee80-6a6d-4b97-be9d-728a85a59e5b-model-cache\") pod \"test-model-3acfdb-kserve-79856cb545-b29mk\" (UID: \"13dfee80-6a6d-4b97-be9d-728a85a59e5b\") " pod="ai-tenant-e2e-inf-b-3acfdb/test-model-3acfdb-kserve-79856cb545-b29mk"
Oct 30 22:41:39.803244 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:41:39.803225    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13dfee80-6a6d-4b97-be9d-728a85a59e5b-home\") pod \"test-model-3acfdb-kserve-79856cb545-b29mk\" (UID: \"13dfee80-6a6d-4b97-be9d-728a85a59e5b\") " pod="ai-tenant-e2e-inf-b-3acfdb/test-model-3acfdb-kserve-79856cb545-b29mk"
Oct 30 22:41:39.803322 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:41:39.803307    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2mgv\" (UniqueName: \"kubernetes.io/projected/13dfee80-6a6d-4b97-be9d-728a85a59e5b-kube-api-access-l2mgv\") pod \"test-model-3acfdb-kserve-79856cb545-b29mk\" (UID: \"13dfee80-6a6d-4b97-be9d-728a85a59e5b\") " pod="ai-tenant-e2e-inf-b-3acfdb/test-model-3acfdb-kserve-79856cb545-b29mk"
Oct 30 22:41:39.803385 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:41:39.803373    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13dfee80-6a6d-4b97-be9d-728a85a59e5b-tmp-dir\") pod \"test-model-3acfdb-kserve-79856cb545-b29mk\" (UID: \"13dfee80-6a6d-4b97-be9d-728a85a59e5b\") " pod="ai-tenant-e2e-inf-b-3acfdb/test-model-3acfdb-kserve-79856cb545-b29mk"
Oct 30 22:41:39.905350 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:41:39.905302    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13dfee80-6a6d-4b97-be9d-728a85a59e5b-tmp-dir\") pod \"test-model-3acfdb-kserve-79856cb545-b29mk\" (UID: \"13dfee80-6a6d-4b97-be9d-728a85a59e5b\") " pod="ai-tenant-e2e-inf-b-3acfdb/test-model-3acfdb-kserve-79856cb545-b29mk"
Oct 30 22:41:39.905523 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:41:39.905413    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13dfee80-6a6d-4b97-be9d-728a85a59e5b-tls-certs\") pod \"test-model-3acfdb-kserve-79856cb545-b29mk\" (UID: \"13dfee80-6a6d-4b97-be9d-728a85a59e5b\") " pod="ai-tenant-e2e-inf-b-3acfdb/test-model-3acfdb-kserve-79856cb545-b29mk"
Oct 30 22:41:39.905523 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:41:39.905459    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13dfee80-6a6d-4b97-be9d-728a85a59e5b-dshm\") pod \"test-model-3acfdb-kserve-79856cb545-b29mk\" (UID: \"13dfee80-6a6d-4b97-be9d-728a85a59e5b\") " pod="ai-tenant-e2e-inf-b-3acfdb/test-model-3acfdb-kserve-79856cb545-b29mk"
Oct 30 22:41:39.905648 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:41:39.905556    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13dfee80-6a6d-4b97-be9d-728a85a59e5b-model-cache\") pod \"test-model-3acfdb-kserve-79856cb545-b29mk\" (UID: \"13dfee80-6a6d-4b97-be9d-728a85a59e5b\") " pod="ai-tenant-e2e-inf-b-3acfdb/test-model-3acfdb-kserve-79856cb545-b29mk"
Oct 30 22:41:39.905704 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:41:39.905654    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13dfee80-6a6d-4b97-be9d-728a85a59e5b-home\") pod \"test-model-3acfdb-kserve-79856cb545-b29mk\" (UID: \"13dfee80-6a6d-4b97-be9d-728a85a59e5b\") " pod="ai-tenant-e2e-inf-b-3acfdb/test-model-3acfdb-kserve-79856cb545-b29mk"
Oct 30 22:41:39.905802 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:41:39.905778    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13dfee80-6a6d-4b97-be9d-728a85a59e5b-tmp-dir\") pod \"test-model-3acfdb-kserve-79856cb545-b29mk\" (UID: \"13dfee80-6a6d-4b97-be9d-728a85a59e5b\") " pod="ai-tenant-e2e-inf-b-3acfdb/test-model-3acfdb-kserve-79856cb545-b29mk"
Oct 30 22:41:39.905867 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:41:39.905781    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2mgv\" (UniqueName: \"kubernetes.io/projected/13dfee80-6a6d-4b97-be9d-728a85a59e5b-kube-api-access-l2mgv\") pod \"test-model-3acfdb-kserve-79856cb545-b29mk\" (UID: \"13dfee80-6a6d-4b97-be9d-728a85a59e5b\") " pod="ai-tenant-e2e-inf-b-3acfdb/test-model-3acfdb-kserve-79856cb545-b29mk"
Oct 30 22:41:39.906257 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:41:39.906227    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13dfee80-6a6d-4b97-be9d-728a85a59e5b-model-cache\") pod \"test-model-3acfdb-kserve-79856cb545-b29mk\" (UID: \"13dfee80-6a6d-4b97-be9d-728a85a59e5b\") " pod="ai-tenant-e2e-inf-b-3acfdb/test-model-3acfdb-kserve-79856cb545-b29mk"
Oct 30 22:41:39.906257 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:41:39.906250    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13dfee80-6a6d-4b97-be9d-728a85a59e5b-home\") pod \"test-model-3acfdb-kserve-79856cb545-b29mk\" (UID: \"13dfee80-6a6d-4b97-be9d-728a85a59e5b\") " pod="ai-tenant-e2e-inf-b-3acfdb/test-model-3acfdb-kserve-79856cb545-b29mk"
Oct 30 22:41:39.908202 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:41:39.908174    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13dfee80-6a6d-4b97-be9d-728a85a59e5b-tls-certs\") pod \"test-model-3acfdb-kserve-79856cb545-b29mk\" (UID: \"13dfee80-6a6d-4b97-be9d-728a85a59e5b\") " pod="ai-tenant-e2e-inf-b-3acfdb/test-model-3acfdb-kserve-79856cb545-b29mk"
Oct 30 22:41:39.908541 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:41:39.908521    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13dfee80-6a6d-4b97-be9d-728a85a59e5b-dshm\") pod \"test-model-3acfdb-kserve-79856cb545-b29mk\" (UID: \"13dfee80-6a6d-4b97-be9d-728a85a59e5b\") " pod="ai-tenant-e2e-inf-b-3acfdb/test-model-3acfdb-kserve-79856cb545-b29mk"
Oct 30 22:41:39.925962 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:41:39.925936    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2mgv\" (UniqueName: \"kubernetes.io/projected/13dfee80-6a6d-4b97-be9d-728a85a59e5b-kube-api-access-l2mgv\") pod \"test-model-3acfdb-kserve-79856cb545-b29mk\" (UID: \"13dfee80-6a6d-4b97-be9d-728a85a59e5b\") " pod="ai-tenant-e2e-inf-b-3acfdb/test-model-3acfdb-kserve-79856cb545-b29mk"
Oct 30 22:41:40.073108 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:41:40.072722    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-3acfdb/test-model-3acfdb-kserve-79856cb545-b29mk"
Oct 30 22:41:40.408444 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:41:40.408391    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-3acfdb/test-model-3acfdb-kserve-79856cb545-b29mk"]
Oct 30 22:41:40.415491 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:41:40.415460    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod13dfee80_6a6d_4b97_be9d_728a85a59e5b.slice/crio-7184f8efcaf96b290014d3cd8ddaf61a48fe59a4e7bab29d45e2953332cee526 WatchSource:0}: Error finding container 7184f8efcaf96b290014d3cd8ddaf61a48fe59a4e7bab29d45e2953332cee526: Status 404 returned error can't find the container with id 7184f8efcaf96b290014d3cd8ddaf61a48fe59a4e7bab29d45e2953332cee526
Oct 30 22:41:41.280826 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:41:41.280789    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-3acfdb/test-model-3acfdb-kserve-79856cb545-b29mk" event={"ID":"13dfee80-6a6d-4b97-be9d-728a85a59e5b","Type":"ContainerStarted","Data":"0ffeeda2858fda9dd9a7a262cede77cd24788736d17a4dccb164492c3773d640"}
Oct 30 22:41:41.281199 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:41:41.280837    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-3acfdb/test-model-3acfdb-kserve-79856cb545-b29mk" event={"ID":"13dfee80-6a6d-4b97-be9d-728a85a59e5b","Type":"ContainerStarted","Data":"7184f8efcaf96b290014d3cd8ddaf61a48fe59a4e7bab29d45e2953332cee526"}
Oct 30 22:41:41.323425 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:41:41.323367    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-3acfdb/test-model-3acfdb-kserve-79856cb545-b29mk" podStartSLOduration=2.3233536089999998 podStartE2EDuration="2.323353609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:41:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:41:41.321705615 +0000 UTC m=+2085.247631821" watchObservedRunningTime="2026-10-30 22:41:41.323353609 +0000 UTC m=+2085.249279814"
Oct 30 22:41:42.073407 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:41:42.073377    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-3acfdb/test-model-3acfdb-kserve-79856cb545-b29mk"
Oct 30 22:41:50.073957 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:41:50.073922    4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-3acfdb/test-model-3acfdb-kserve-79856cb545-b29mk"
Oct 30 22:41:50.086156 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:41:50.086132    4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-3acfdb/test-model-3acfdb-kserve-79856cb545-b29mk"
Oct 30 22:41:50.361424 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:41:50.361350    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-3acfdb/test-model-3acfdb-kserve-79856cb545-b29mk"
Oct 30 22:41:51.972408 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:41:51.972376    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-6922d3/test-model-6922d3-kserve-65bc85fbc-tcdmn"]
Oct 30 22:41:56.281158 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:41:56.281112    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 22:41:56.910303 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:41:56.910275    4477 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jcf8l_015f218d-bf1f-4e7e-bb5d-b805af003246/ovn-acl-logging/0.log"
Oct 30 22:41:56.958682 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:41:56.958658    4477 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jcf8l_015f218d-bf1f-4e7e-bb5d-b805af003246/ovn-acl-logging/0.log"
Oct 30 22:42:06.996681 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:06.996642    4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-6922d3/test-model-6922d3-kserve-65bc85fbc-tcdmn" podUID="cccc800e-68e3-40ff-b7a0-29f5a777e83b" containerName="main" containerID="cri-o://4496852a9575c119abd9627773fbc2798fd3ebe6417658082676189632951414" gracePeriod=45
Oct 30 22:42:07.284710 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:07.284685    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-6922d3/test-model-6922d3-kserve-65bc85fbc-tcdmn"
Oct 30 22:42:07.324498 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:07.324471    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cccc800e-68e3-40ff-b7a0-29f5a777e83b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cccc800e-68e3-40ff-b7a0-29f5a777e83b-model-cache\") pod \"cccc800e-68e3-40ff-b7a0-29f5a777e83b\" (UID: \"cccc800e-68e3-40ff-b7a0-29f5a777e83b\") "
Oct 30 22:42:07.324668 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:07.324556    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cccc800e-68e3-40ff-b7a0-29f5a777e83b-home\" (UniqueName: \"kubernetes.io/empty-dir/cccc800e-68e3-40ff-b7a0-29f5a777e83b-home\") pod \"cccc800e-68e3-40ff-b7a0-29f5a777e83b\" (UID: \"cccc800e-68e3-40ff-b7a0-29f5a777e83b\") "
Oct 30 22:42:07.324668 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:07.324599    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cccc800e-68e3-40ff-b7a0-29f5a777e83b-kube-api-access-l5vrx\" (UniqueName: \"kubernetes.io/projected/cccc800e-68e3-40ff-b7a0-29f5a777e83b-kube-api-access-l5vrx\") pod \"cccc800e-68e3-40ff-b7a0-29f5a777e83b\" (UID: \"cccc800e-68e3-40ff-b7a0-29f5a777e83b\") "
Oct 30 22:42:07.324815 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:07.324716    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cccc800e-68e3-40ff-b7a0-29f5a777e83b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cccc800e-68e3-40ff-b7a0-29f5a777e83b-dshm\") pod \"cccc800e-68e3-40ff-b7a0-29f5a777e83b\" (UID: \"cccc800e-68e3-40ff-b7a0-29f5a777e83b\") "
Oct 30 22:42:07.324815 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:07.324760    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cccc800e-68e3-40ff-b7a0-29f5a777e83b-model-cache" pod "cccc800e-68e3-40ff-b7a0-29f5a777e83b" (UID: "cccc800e-68e3-40ff-b7a0-29f5a777e83b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:42:07.324920 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:07.324828    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cccc800e-68e3-40ff-b7a0-29f5a777e83b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cccc800e-68e3-40ff-b7a0-29f5a777e83b-tmp-dir\") pod \"cccc800e-68e3-40ff-b7a0-29f5a777e83b\" (UID: \"cccc800e-68e3-40ff-b7a0-29f5a777e83b\") "
Oct 30 22:42:07.324970 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:07.324943    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cccc800e-68e3-40ff-b7a0-29f5a777e83b-home" pod "cccc800e-68e3-40ff-b7a0-29f5a777e83b" (UID: "cccc800e-68e3-40ff-b7a0-29f5a777e83b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:42:07.325029 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:07.324989    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cccc800e-68e3-40ff-b7a0-29f5a777e83b-tls-certs\" (UniqueName: \"kubernetes.io/secret/cccc800e-68e3-40ff-b7a0-29f5a777e83b-tls-certs\") pod \"cccc800e-68e3-40ff-b7a0-29f5a777e83b\" (UID: \"cccc800e-68e3-40ff-b7a0-29f5a777e83b\") "
Oct 30 22:42:07.325473 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:07.325429    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cccc800e-68e3-40ff-b7a0-29f5a777e83b-tmp-dir" pod "cccc800e-68e3-40ff-b7a0-29f5a777e83b" (UID: "cccc800e-68e3-40ff-b7a0-29f5a777e83b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:42:07.325668 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:07.325643    4477 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cccc800e-68e3-40ff-b7a0-29f5a777e83b-model-cache\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:42:07.325746 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:07.325672    4477 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cccc800e-68e3-40ff-b7a0-29f5a777e83b-home\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:42:07.325746 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:07.325686    4477 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cccc800e-68e3-40ff-b7a0-29f5a777e83b-tmp-dir\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:42:07.327897 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:07.327860    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cccc800e-68e3-40ff-b7a0-29f5a777e83b-dshm" pod "cccc800e-68e3-40ff-b7a0-29f5a777e83b" (UID: "cccc800e-68e3-40ff-b7a0-29f5a777e83b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:42:07.328259 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:07.328236    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cccc800e-68e3-40ff-b7a0-29f5a777e83b-tls-certs" pod "cccc800e-68e3-40ff-b7a0-29f5a777e83b" (UID: "cccc800e-68e3-40ff-b7a0-29f5a777e83b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:42:07.329022 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:07.328985    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cccc800e-68e3-40ff-b7a0-29f5a777e83b-kube-api-access-l5vrx" pod "cccc800e-68e3-40ff-b7a0-29f5a777e83b" (UID: "cccc800e-68e3-40ff-b7a0-29f5a777e83b"). InnerVolumeSpecName "kube-api-access-l5vrx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:42:07.427226 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:07.427193    4477 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cccc800e-68e3-40ff-b7a0-29f5a777e83b-tls-certs\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:42:07.427226 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:07.427224    4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5vrx\" (UniqueName: \"kubernetes.io/projected/cccc800e-68e3-40ff-b7a0-29f5a777e83b-kube-api-access-l5vrx\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:42:07.427226 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:07.427233    4477 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cccc800e-68e3-40ff-b7a0-29f5a777e83b-dshm\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:42:07.467019 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:07.466988    4477 generic.go:356] "Generic (PLEG): container finished" podID="cccc800e-68e3-40ff-b7a0-29f5a777e83b" containerID="4496852a9575c119abd9627773fbc2798fd3ebe6417658082676189632951414" exitCode=0
Oct 30 22:42:07.467176 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:07.467029    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-6922d3/test-model-6922d3-kserve-65bc85fbc-tcdmn"
Oct 30 22:42:07.467176 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:07.467066    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-6922d3/test-model-6922d3-kserve-65bc85fbc-tcdmn" event={"ID":"cccc800e-68e3-40ff-b7a0-29f5a777e83b","Type":"ContainerDied","Data":"4496852a9575c119abd9627773fbc2798fd3ebe6417658082676189632951414"}
Oct 30 22:42:07.467176 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:07.467111    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-6922d3/test-model-6922d3-kserve-65bc85fbc-tcdmn" event={"ID":"cccc800e-68e3-40ff-b7a0-29f5a777e83b","Type":"ContainerDied","Data":"ff7385b77733623a6cce6363ae776f4884835d53cade3076d52ebb19ca5360a5"}
Oct 30 22:42:07.467176 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:07.467128    4477 scope.go:122] "RemoveContainer" containerID="4496852a9575c119abd9627773fbc2798fd3ebe6417658082676189632951414"
Oct 30 22:42:07.479213 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:07.479174    4477 scope.go:122] "RemoveContainer" containerID="4496852a9575c119abd9627773fbc2798fd3ebe6417658082676189632951414"
Oct 30 22:42:07.479464 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:42:07.479442    4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4496852a9575c119abd9627773fbc2798fd3ebe6417658082676189632951414\": container with ID starting with 4496852a9575c119abd9627773fbc2798fd3ebe6417658082676189632951414 not found: ID does not exist" containerID="4496852a9575c119abd9627773fbc2798fd3ebe6417658082676189632951414"
Oct 30 22:42:07.479557 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:07.479472    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4496852a9575c119abd9627773fbc2798fd3ebe6417658082676189632951414"} err="failed to get container status \"4496852a9575c119abd9627773fbc2798fd3ebe6417658082676189632951414\": rpc error: code = NotFound desc = could not find container \"4496852a9575c119abd9627773fbc2798fd3ebe6417658082676189632951414\": container with ID starting with 4496852a9575c119abd9627773fbc2798fd3ebe6417658082676189632951414 not found: ID does not exist"
Oct 30 22:42:07.518848 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:07.518794    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-6922d3/test-model-6922d3-kserve-65bc85fbc-tcdmn"]
Oct 30 22:42:07.523009 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:07.522991    4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-6922d3/test-model-6922d3-kserve-65bc85fbc-tcdmn"]
Oct 30 22:42:08.195314 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:08.195273    4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cccc800e-68e3-40ff-b7a0-29f5a777e83b" path="/var/lib/kubelet/pods/cccc800e-68e3-40ff-b7a0-29f5a777e83b/volumes"
Oct 30 22:42:15.173934 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:15.173846    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-f9d87b/test-model-f9d87b-kserve-58b9dc6999-ttktm"]
Oct 30 22:42:26.199285 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:26.199254    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-kvtw4"]
Oct 30 22:42:26.199699 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:26.199460    4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-kvtw4" podUID="0a99ab39-789a-4cbf-8db5-5e85bfe485c2" containerName="istio-proxy" containerID="cri-o://df69e97adfaae616164d5729ef54b4bb176685252fac4c457de03fd8bb03bec9" gracePeriod=30
Oct 30 22:42:26.267764 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:42:26.267728    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 22:42:27.588754 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:27.588716    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6922d3-544f4f95d4-8psb5"]
Oct 30 22:42:27.589299 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:27.588963    4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6922d3-544f4f95d4-8psb5" podUID="4e931bc1-b014-48ed-beaa-e82a64000d6a" containerName="maas-api" containerID="cri-o://d6fb0bb9b68e9775456b332e8dfc3789151e4d33e1f4bed1231e59c92a8a9255" gracePeriod=30
Oct 30 22:42:28.229437 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:28.226702    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6922d3-544f4f95d4-8psb5"
Oct 30 22:42:28.240816 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:28.240783    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e931bc1-b014-48ed-beaa-e82a64000d6a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e931bc1-b014-48ed-beaa-e82a64000d6a-openshift-service-ca\") pod \"4e931bc1-b014-48ed-beaa-e82a64000d6a\" (UID: \"4e931bc1-b014-48ed-beaa-e82a64000d6a\") "
Oct 30 22:42:28.241009 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:28.240992    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e931bc1-b014-48ed-beaa-e82a64000d6a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e931bc1-b014-48ed-beaa-e82a64000d6a-metrics-certs\") pod \"4e931bc1-b014-48ed-beaa-e82a64000d6a\" (UID: \"4e931bc1-b014-48ed-beaa-e82a64000d6a\") "
Oct 30 22:42:28.241162 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:28.241031    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e931bc1-b014-48ed-beaa-e82a64000d6a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e931bc1-b014-48ed-beaa-e82a64000d6a-maas-api-tls\") pod \"4e931bc1-b014-48ed-beaa-e82a64000d6a\" (UID: \"4e931bc1-b014-48ed-beaa-e82a64000d6a\") "
Oct 30 22:42:28.241162 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:28.241102    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e931bc1-b014-48ed-beaa-e82a64000d6a-kube-api-access-j2wf6\" (UniqueName: \"kubernetes.io/projected/4e931bc1-b014-48ed-beaa-e82a64000d6a-kube-api-access-j2wf6\") pod \"4e931bc1-b014-48ed-beaa-e82a64000d6a\" (UID: \"4e931bc1-b014-48ed-beaa-e82a64000d6a\") "
Oct 30 22:42:28.241288 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:28.241199    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e931bc1-b014-48ed-beaa-e82a64000d6a-openshift-service-ca" pod "4e931bc1-b014-48ed-beaa-e82a64000d6a" (UID: "4e931bc1-b014-48ed-beaa-e82a64000d6a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:42:28.241739 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:28.241716    4477 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e931bc1-b014-48ed-beaa-e82a64000d6a-openshift-service-ca\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:42:28.243507 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:28.243467    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e931bc1-b014-48ed-beaa-e82a64000d6a-metrics-certs" pod "4e931bc1-b014-48ed-beaa-e82a64000d6a" (UID: "4e931bc1-b014-48ed-beaa-e82a64000d6a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:42:28.243689 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:28.243664    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e931bc1-b014-48ed-beaa-e82a64000d6a-kube-api-access-j2wf6" pod "4e931bc1-b014-48ed-beaa-e82a64000d6a" (UID: "4e931bc1-b014-48ed-beaa-e82a64000d6a"). InnerVolumeSpecName "kube-api-access-j2wf6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:42:28.243828 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:28.243806    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e931bc1-b014-48ed-beaa-e82a64000d6a-maas-api-tls" pod "4e931bc1-b014-48ed-beaa-e82a64000d6a" (UID: "4e931bc1-b014-48ed-beaa-e82a64000d6a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:42:28.344861 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:28.343878    4477 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e931bc1-b014-48ed-beaa-e82a64000d6a-metrics-certs\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:42:28.344861 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:28.343926    4477 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e931bc1-b014-48ed-beaa-e82a64000d6a-maas-api-tls\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:42:28.344861 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:28.343942    4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2wf6\" (UniqueName: \"kubernetes.io/projected/4e931bc1-b014-48ed-beaa-e82a64000d6a-kube-api-access-j2wf6\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:42:28.637858 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:28.637812    4477 generic.go:356] "Generic (PLEG): container finished" podID="4e931bc1-b014-48ed-beaa-e82a64000d6a" containerID="d6fb0bb9b68e9775456b332e8dfc3789151e4d33e1f4bed1231e59c92a8a9255" exitCode=0
Oct 30 22:42:28.638454 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:28.637922    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6922d3-544f4f95d4-8psb5"
Oct 30 22:42:28.638454 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:28.637961    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6922d3-544f4f95d4-8psb5" event={"ID":"4e931bc1-b014-48ed-beaa-e82a64000d6a","Type":"ContainerDied","Data":"d6fb0bb9b68e9775456b332e8dfc3789151e4d33e1f4bed1231e59c92a8a9255"}
Oct 30 22:42:28.638454 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:28.638020    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6922d3-544f4f95d4-8psb5" event={"ID":"4e931bc1-b014-48ed-beaa-e82a64000d6a","Type":"ContainerDied","Data":"5d17fdcd82691b83abe1909fd536e22c6388addec35045756f4357ebcbe3870d"}
Oct 30 22:42:28.638454 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:28.638089    4477 scope.go:122] "RemoveContainer" containerID="d6fb0bb9b68e9775456b332e8dfc3789151e4d33e1f4bed1231e59c92a8a9255"
Oct 30 22:42:28.651697 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:28.651675    4477 scope.go:122] "RemoveContainer" containerID="d6fb0bb9b68e9775456b332e8dfc3789151e4d33e1f4bed1231e59c92a8a9255"
Oct 30 22:42:28.651987 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:42:28.651965    4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6fb0bb9b68e9775456b332e8dfc3789151e4d33e1f4bed1231e59c92a8a9255\": container with ID starting with d6fb0bb9b68e9775456b332e8dfc3789151e4d33e1f4bed1231e59c92a8a9255 not found: ID does not exist" containerID="d6fb0bb9b68e9775456b332e8dfc3789151e4d33e1f4bed1231e59c92a8a9255"
Oct 30 22:42:28.652125 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:28.651994    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6fb0bb9b68e9775456b332e8dfc3789151e4d33e1f4bed1231e59c92a8a9255"} err="failed to get container status \"d6fb0bb9b68e9775456b332e8dfc3789151e4d33e1f4bed1231e59c92a8a9255\": rpc error: code = NotFound desc = could not find container \"d6fb0bb9b68e9775456b332e8dfc3789151e4d33e1f4bed1231e59c92a8a9255\": container with ID starting with d6fb0bb9b68e9775456b332e8dfc3789151e4d33e1f4bed1231e59c92a8a9255 not found: ID does not exist"
Oct 30 22:42:28.692114 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:28.692076    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6922d3-544f4f95d4-8psb5"]
Oct 30 22:42:28.703536 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:28.703451    4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6922d3-544f4f95d4-8psb5"]
Oct 30 22:42:30.196138 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:30.196100    4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e931bc1-b014-48ed-beaa-e82a64000d6a" path="/var/lib/kubelet/pods/4e931bc1-b014-48ed-beaa-e82a64000d6a/volumes"
Oct 30 22:42:30.209881 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:30.209837    4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-f9d87b/test-model-f9d87b-kserve-58b9dc6999-ttktm" podUID="5e7cf3ef-0cd6-415f-8fb5-b806fd818c77" containerName="main" containerID="cri-o://626b03c860aae36166a7292849a84b26fc7acfc9429912380442dfa01b3f04f0" gracePeriod=45
Oct 30 22:42:30.496508 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:30.496484    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f9d87b/test-model-f9d87b-kserve-58b9dc6999-ttktm"
Oct 30 22:42:30.575159 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:30.575120    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e7cf3ef-0cd6-415f-8fb5-b806fd818c77-kube-api-access-q2vbl\" (UniqueName: \"kubernetes.io/projected/5e7cf3ef-0cd6-415f-8fb5-b806fd818c77-kube-api-access-q2vbl\") pod \"5e7cf3ef-0cd6-415f-8fb5-b806fd818c77\" (UID: \"5e7cf3ef-0cd6-415f-8fb5-b806fd818c77\") "
Oct 30 22:42:30.575330 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:30.575309    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e7cf3ef-0cd6-415f-8fb5-b806fd818c77-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e7cf3ef-0cd6-415f-8fb5-b806fd818c77-dshm\") pod \"5e7cf3ef-0cd6-415f-8fb5-b806fd818c77\" (UID: \"5e7cf3ef-0cd6-415f-8fb5-b806fd818c77\") "
Oct 30 22:42:30.575443 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:30.575425    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e7cf3ef-0cd6-415f-8fb5-b806fd818c77-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e7cf3ef-0cd6-415f-8fb5-b806fd818c77-model-cache\") pod \"5e7cf3ef-0cd6-415f-8fb5-b806fd818c77\" (UID: \"5e7cf3ef-0cd6-415f-8fb5-b806fd818c77\") "
Oct 30 22:42:30.575529 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:30.575516    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e7cf3ef-0cd6-415f-8fb5-b806fd818c77-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e7cf3ef-0cd6-415f-8fb5-b806fd818c77-tmp-dir\") pod \"5e7cf3ef-0cd6-415f-8fb5-b806fd818c77\" (UID: \"5e7cf3ef-0cd6-415f-8fb5-b806fd818c77\") "
Oct 30 22:42:30.575612 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:30.575597    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e7cf3ef-0cd6-415f-8fb5-b806fd818c77-tls-certs\" (UniqueName: \"kubernetes.io/secret/5e7cf3ef-0cd6-415f-8fb5-b806fd818c77-tls-certs\") pod \"5e7cf3ef-0cd6-415f-8fb5-b806fd818c77\" (UID: \"5e7cf3ef-0cd6-415f-8fb5-b806fd818c77\") "
Oct 30 22:42:30.575739 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:30.575723    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e7cf3ef-0cd6-415f-8fb5-b806fd818c77-home\" (UniqueName: \"kubernetes.io/empty-dir/5e7cf3ef-0cd6-415f-8fb5-b806fd818c77-home\") pod \"5e7cf3ef-0cd6-415f-8fb5-b806fd818c77\" (UID: \"5e7cf3ef-0cd6-415f-8fb5-b806fd818c77\") "
Oct 30 22:42:30.575820 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:30.575745    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e7cf3ef-0cd6-415f-8fb5-b806fd818c77-model-cache" pod "5e7cf3ef-0cd6-415f-8fb5-b806fd818c77" (UID: "5e7cf3ef-0cd6-415f-8fb5-b806fd818c77"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:42:30.575970 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:30.575946    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e7cf3ef-0cd6-415f-8fb5-b806fd818c77-tmp-dir" pod "5e7cf3ef-0cd6-415f-8fb5-b806fd818c77" (UID: "5e7cf3ef-0cd6-415f-8fb5-b806fd818c77"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:42:30.576573 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:30.576390    4477 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e7cf3ef-0cd6-415f-8fb5-b806fd818c77-tmp-dir\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:42:30.576573 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:30.576413    4477 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e7cf3ef-0cd6-415f-8fb5-b806fd818c77-model-cache\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:42:30.576573 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:30.576512    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e7cf3ef-0cd6-415f-8fb5-b806fd818c77-home" pod "5e7cf3ef-0cd6-415f-8fb5-b806fd818c77" (UID: "5e7cf3ef-0cd6-415f-8fb5-b806fd818c77"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:42:30.578153 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:30.578127    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e7cf3ef-0cd6-415f-8fb5-b806fd818c77-kube-api-access-q2vbl" pod "5e7cf3ef-0cd6-415f-8fb5-b806fd818c77" (UID: "5e7cf3ef-0cd6-415f-8fb5-b806fd818c77"). InnerVolumeSpecName "kube-api-access-q2vbl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:42:30.578261 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:30.578179    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e7cf3ef-0cd6-415f-8fb5-b806fd818c77-dshm" pod "5e7cf3ef-0cd6-415f-8fb5-b806fd818c77" (UID: "5e7cf3ef-0cd6-415f-8fb5-b806fd818c77"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:42:30.578752 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:30.578728    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e7cf3ef-0cd6-415f-8fb5-b806fd818c77-tls-certs" pod "5e7cf3ef-0cd6-415f-8fb5-b806fd818c77" (UID: "5e7cf3ef-0cd6-415f-8fb5-b806fd818c77"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:42:30.655692 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:30.655664    4477 generic.go:356] "Generic (PLEG): container finished" podID="5e7cf3ef-0cd6-415f-8fb5-b806fd818c77" containerID="626b03c860aae36166a7292849a84b26fc7acfc9429912380442dfa01b3f04f0" exitCode=0
Oct 30 22:42:30.655793 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:30.655731    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f9d87b/test-model-f9d87b-kserve-58b9dc6999-ttktm"
Oct 30 22:42:30.655793 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:30.655749    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f9d87b/test-model-f9d87b-kserve-58b9dc6999-ttktm" event={"ID":"5e7cf3ef-0cd6-415f-8fb5-b806fd818c77","Type":"ContainerDied","Data":"626b03c860aae36166a7292849a84b26fc7acfc9429912380442dfa01b3f04f0"}
Oct 30 22:42:30.655793 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:30.655792    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f9d87b/test-model-f9d87b-kserve-58b9dc6999-ttktm" event={"ID":"5e7cf3ef-0cd6-415f-8fb5-b806fd818c77","Type":"ContainerDied","Data":"d6a84291848de127dd2793fa303e0060ba8eb2b72cb7695c9345b3f5d71756cb"}
Oct 30 22:42:30.655909 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:30.655808    4477 scope.go:122] "RemoveContainer" containerID="626b03c860aae36166a7292849a84b26fc7acfc9429912380442dfa01b3f04f0"
Oct 30 22:42:30.673290 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:30.673260    4477 scope.go:122] "RemoveContainer" containerID="626b03c860aae36166a7292849a84b26fc7acfc9429912380442dfa01b3f04f0"
Oct 30 22:42:30.673594 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:42:30.673569    4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"626b03c860aae36166a7292849a84b26fc7acfc9429912380442dfa01b3f04f0\": container with ID starting with 626b03c860aae36166a7292849a84b26fc7acfc9429912380442dfa01b3f04f0 not found: ID does not exist" containerID="626b03c860aae36166a7292849a84b26fc7acfc9429912380442dfa01b3f04f0"
Oct 30 22:42:30.673680 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:30.673604    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"626b03c860aae36166a7292849a84b26fc7acfc9429912380442dfa01b3f04f0"} err="failed to get container status \"626b03c860aae36166a7292849a84b26fc7acfc9429912380442dfa01b3f04f0\": rpc error: code = NotFound desc = could not find container \"626b03c860aae36166a7292849a84b26fc7acfc9429912380442dfa01b3f04f0\": container with ID starting with 626b03c860aae36166a7292849a84b26fc7acfc9429912380442dfa01b3f04f0 not found: ID does not exist"
Oct 30 22:42:30.677284 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:30.677263    4477 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e7cf3ef-0cd6-415f-8fb5-b806fd818c77-home\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:42:30.677363 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:30.677304    4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2vbl\" (UniqueName: \"kubernetes.io/projected/5e7cf3ef-0cd6-415f-8fb5-b806fd818c77-kube-api-access-q2vbl\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:42:30.677363 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:30.677314    4477 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e7cf3ef-0cd6-415f-8fb5-b806fd818c77-dshm\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:42:30.677363 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:30.677323    4477 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e7cf3ef-0cd6-415f-8fb5-b806fd818c77-tls-certs\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:42:30.710167 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:30.710147    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-f9d87b/test-model-f9d87b-kserve-58b9dc6999-ttktm"]
Oct 30 22:42:30.713818 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:30.713772    4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-f9d87b/test-model-f9d87b-kserve-58b9dc6999-ttktm"]
Oct 30 22:42:31.035138 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:31.034647    4477 patch_prober.go:29] interesting pod/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-kvtw4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:42:31.035138 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:31.034729    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-kvtw4" podUID="0a99ab39-789a-4cbf-8db5-5e85bfe485c2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:42:31.490298 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:31.490275    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-kvtw4"
Oct 30 22:42:31.589898 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:31.589829    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a99ab39-789a-4cbf-8db5-5e85bfe485c2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a99ab39-789a-4cbf-8db5-5e85bfe485c2-workload-certs\") pod \"0a99ab39-789a-4cbf-8db5-5e85bfe485c2\" (UID: \"0a99ab39-789a-4cbf-8db5-5e85bfe485c2\") "
Oct 30 22:42:31.589898 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:31.589870    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a99ab39-789a-4cbf-8db5-5e85bfe485c2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a99ab39-789a-4cbf-8db5-5e85bfe485c2-workload-socket\") pod \"0a99ab39-789a-4cbf-8db5-5e85bfe485c2\" (UID: \"0a99ab39-789a-4cbf-8db5-5e85bfe485c2\") "
Oct 30 22:42:31.589898 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:31.589898    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a99ab39-789a-4cbf-8db5-5e85bfe485c2-kube-api-access-hgc78\" (UniqueName: \"kubernetes.io/projected/0a99ab39-789a-4cbf-8db5-5e85bfe485c2-kube-api-access-hgc78\") pod \"0a99ab39-789a-4cbf-8db5-5e85bfe485c2\" (UID: \"0a99ab39-789a-4cbf-8db5-5e85bfe485c2\") "
Oct 30 22:42:31.590165 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:31.589929    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0a99ab39-789a-4cbf-8db5-5e85bfe485c2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a99ab39-789a-4cbf-8db5-5e85bfe485c2-istio-podinfo\") pod \"0a99ab39-789a-4cbf-8db5-5e85bfe485c2\" (UID: \"0a99ab39-789a-4cbf-8db5-5e85bfe485c2\") "
Oct 30 22:42:31.590165 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:31.589954    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a99ab39-789a-4cbf-8db5-5e85bfe485c2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a99ab39-789a-4cbf-8db5-5e85bfe485c2-istio-data\") pod \"0a99ab39-789a-4cbf-8db5-5e85bfe485c2\" (UID: \"0a99ab39-789a-4cbf-8db5-5e85bfe485c2\") "
Oct 30 22:42:31.590165 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:31.589998    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a99ab39-789a-4cbf-8db5-5e85bfe485c2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a99ab39-789a-4cbf-8db5-5e85bfe485c2-credential-socket\") pod \"0a99ab39-789a-4cbf-8db5-5e85bfe485c2\" (UID: \"0a99ab39-789a-4cbf-8db5-5e85bfe485c2\") "
Oct 30 22:42:31.590165 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:31.590068    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a99ab39-789a-4cbf-8db5-5e85bfe485c2-istio-token\" (UniqueName: \"kubernetes.io/projected/0a99ab39-789a-4cbf-8db5-5e85bfe485c2-istio-token\") pod \"0a99ab39-789a-4cbf-8db5-5e85bfe485c2\" (UID: \"0a99ab39-789a-4cbf-8db5-5e85bfe485c2\") "
Oct 30 22:42:31.590165 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:31.590142    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a99ab39-789a-4cbf-8db5-5e85bfe485c2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a99ab39-789a-4cbf-8db5-5e85bfe485c2-istio-envoy\") pod \"0a99ab39-789a-4cbf-8db5-5e85bfe485c2\" (UID: \"0a99ab39-789a-4cbf-8db5-5e85bfe485c2\") "
Oct 30 22:42:31.590165 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:31.590148    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a99ab39-789a-4cbf-8db5-5e85bfe485c2-workload-certs" pod "0a99ab39-789a-4cbf-8db5-5e85bfe485c2" (UID: "0a99ab39-789a-4cbf-8db5-5e85bfe485c2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:42:31.590482 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:31.590171    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a99ab39-789a-4cbf-8db5-5e85bfe485c2-workload-socket" pod "0a99ab39-789a-4cbf-8db5-5e85bfe485c2" (UID: "0a99ab39-789a-4cbf-8db5-5e85bfe485c2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:42:31.590482 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:31.590289    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a99ab39-789a-4cbf-8db5-5e85bfe485c2-istio-data" pod "0a99ab39-789a-4cbf-8db5-5e85bfe485c2" (UID: "0a99ab39-789a-4cbf-8db5-5e85bfe485c2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:42:31.590482 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:31.590408    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a99ab39-789a-4cbf-8db5-5e85bfe485c2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a99ab39-789a-4cbf-8db5-5e85bfe485c2-istiod-ca-cert\") pod \"0a99ab39-789a-4cbf-8db5-5e85bfe485c2\" (UID: \"0a99ab39-789a-4cbf-8db5-5e85bfe485c2\") "
Oct 30 22:42:31.590653 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:31.590580    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a99ab39-789a-4cbf-8db5-5e85bfe485c2-credential-socket" pod "0a99ab39-789a-4cbf-8db5-5e85bfe485c2" (UID: "0a99ab39-789a-4cbf-8db5-5e85bfe485c2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:42:31.590827 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:31.590800    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a99ab39-789a-4cbf-8db5-5e85bfe485c2-istiod-ca-cert" pod "0a99ab39-789a-4cbf-8db5-5e85bfe485c2" (UID: "0a99ab39-789a-4cbf-8db5-5e85bfe485c2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:42:31.591020 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:31.591002    4477 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a99ab39-789a-4cbf-8db5-5e85bfe485c2-istiod-ca-cert\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:42:31.591114 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:31.591021    4477 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a99ab39-789a-4cbf-8db5-5e85bfe485c2-workload-certs\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:42:31.591114 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:31.591030    4477 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a99ab39-789a-4cbf-8db5-5e85bfe485c2-workload-socket\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:42:31.591114 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:31.591051    4477 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a99ab39-789a-4cbf-8db5-5e85bfe485c2-istio-data\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:42:31.591114 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:31.591063    4477 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a99ab39-789a-4cbf-8db5-5e85bfe485c2-credential-socket\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:42:31.592362 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:31.592330    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a99ab39-789a-4cbf-8db5-5e85bfe485c2-kube-api-access-hgc78" pod "0a99ab39-789a-4cbf-8db5-5e85bfe485c2" (UID: "0a99ab39-789a-4cbf-8db5-5e85bfe485c2"). InnerVolumeSpecName "kube-api-access-hgc78". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:42:31.592478 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:31.592414    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a99ab39-789a-4cbf-8db5-5e85bfe485c2-istio-token" pod "0a99ab39-789a-4cbf-8db5-5e85bfe485c2" (UID: "0a99ab39-789a-4cbf-8db5-5e85bfe485c2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:42:31.592715 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:31.592691    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0a99ab39-789a-4cbf-8db5-5e85bfe485c2-istio-podinfo" pod "0a99ab39-789a-4cbf-8db5-5e85bfe485c2" (UID: "0a99ab39-789a-4cbf-8db5-5e85bfe485c2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:42:31.592796 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:31.592719    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a99ab39-789a-4cbf-8db5-5e85bfe485c2-istio-envoy" pod "0a99ab39-789a-4cbf-8db5-5e85bfe485c2" (UID: "0a99ab39-789a-4cbf-8db5-5e85bfe485c2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:42:31.666311 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:31.666277    4477 generic.go:356] "Generic (PLEG): container finished" podID="0a99ab39-789a-4cbf-8db5-5e85bfe485c2" containerID="df69e97adfaae616164d5729ef54b4bb176685252fac4c457de03fd8bb03bec9" exitCode=0
Oct 30 22:42:31.666472 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:31.666326    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-kvtw4"
Oct 30 22:42:31.666472 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:31.666354    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-kvtw4" event={"ID":"0a99ab39-789a-4cbf-8db5-5e85bfe485c2","Type":"ContainerDied","Data":"df69e97adfaae616164d5729ef54b4bb176685252fac4c457de03fd8bb03bec9"}
Oct 30 22:42:31.666472 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:31.666408    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-kvtw4" event={"ID":"0a99ab39-789a-4cbf-8db5-5e85bfe485c2","Type":"ContainerDied","Data":"82c270fdc8137654099e656f24b6b55b29d2703900753f0b26663480ef419789"}
Oct 30 22:42:31.666472 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:31.666431    4477 scope.go:122] "RemoveContainer" containerID="df69e97adfaae616164d5729ef54b4bb176685252fac4c457de03fd8bb03bec9"
Oct 30 22:42:31.677794 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:31.677776    4477 scope.go:122] "RemoveContainer" containerID="df69e97adfaae616164d5729ef54b4bb176685252fac4c457de03fd8bb03bec9"
Oct 30 22:42:31.678114 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:42:31.678093    4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df69e97adfaae616164d5729ef54b4bb176685252fac4c457de03fd8bb03bec9\": container with ID starting with df69e97adfaae616164d5729ef54b4bb176685252fac4c457de03fd8bb03bec9 not found: ID does not exist" containerID="df69e97adfaae616164d5729ef54b4bb176685252fac4c457de03fd8bb03bec9"
Oct 30 22:42:31.678187 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:31.678122    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df69e97adfaae616164d5729ef54b4bb176685252fac4c457de03fd8bb03bec9"} err="failed to get container status \"df69e97adfaae616164d5729ef54b4bb176685252fac4c457de03fd8bb03bec9\": rpc error: code = NotFound desc = could not find container \"df69e97adfaae616164d5729ef54b4bb176685252fac4c457de03fd8bb03bec9\": container with ID starting with df69e97adfaae616164d5729ef54b4bb176685252fac4c457de03fd8bb03bec9 not found: ID does not exist"
Oct 30 22:42:31.692807 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:31.692786    4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgc78\" (UniqueName: \"kubernetes.io/projected/0a99ab39-789a-4cbf-8db5-5e85bfe485c2-kube-api-access-hgc78\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:42:31.692807 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:31.692808    4477 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a99ab39-789a-4cbf-8db5-5e85bfe485c2-istio-podinfo\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:42:31.692934 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:31.692818    4477 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a99ab39-789a-4cbf-8db5-5e85bfe485c2-istio-token\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:42:31.692934 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:31.692825    4477 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a99ab39-789a-4cbf-8db5-5e85bfe485c2-istio-envoy\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:42:31.719713 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:31.719692    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-kvtw4"]
Oct 30 22:42:31.726006 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:31.725980    4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-54114e-openshift-default-7fc467ddd7-kvtw4"]
Oct 30 22:42:32.195012 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:32.194982    4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a99ab39-789a-4cbf-8db5-5e85bfe485c2" path="/var/lib/kubelet/pods/0a99ab39-789a-4cbf-8db5-5e85bfe485c2/volumes"
Oct 30 22:42:32.195579 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:32.195557    4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e7cf3ef-0cd6-415f-8fb5-b806fd818c77" path="/var/lib/kubelet/pods/5e7cf3ef-0cd6-415f-8fb5-b806fd818c77/volumes"
Oct 30 22:42:36.184938 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:36.184903    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-x68sw"]
Oct 30 22:42:36.185436 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:36.185168    4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-x68sw" podUID="ca4b71ad-f474-4db6-a350-8f1da8fe6244" containerName="istio-proxy" containerID="cri-o://64627c9156678f089e05be2217551c32c7a9e7303a7f30a797dc1d24c08235b5" gracePeriod=30
Oct 30 22:42:36.209001 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:36.208968    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-7bhqj"]
Oct 30 22:42:36.209279 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:36.209230    4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-7bhqj" podUID="58693405-8a5b-4841-865d-aa29cc320e4d" containerName="istio-proxy" containerID="cri-o://27913e02cf990f9b7065601c31973768d95c7cfc49c140fb17043f00795551a1" gracePeriod=30
Oct 30 22:42:36.216294 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:36.216269    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-txq8f"]
Oct 30 22:42:36.216480 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:36.216458    4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-txq8f" podUID="95030005-bb27-46a1-9238-503138ca97f6" containerName="istio-proxy" containerID="cri-o://22294cb010b5e5259f33a86d207c642f175cfb7afcf2d391740469eec302ee14" gracePeriod=30
Oct 30 22:42:37.103002 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:37.102969    4477 patch_prober.go:29] interesting pod/e2e-inf-a-6922d3-openshift-default-689b5f95bc-7bhqj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:42:37.103190 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:37.103029    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-7bhqj" podUID="58693405-8a5b-4841-865d-aa29cc320e4d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:42:37.242285 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:37.242247    4477 patch_prober.go:29] interesting pod/e2e-inf-a-6922d3-openshift-default-689b5f95bc-txq8f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:42:37.242668 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:37.242302    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-txq8f" podUID="95030005-bb27-46a1-9238-503138ca97f6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:42:38.499786 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:38.499750    4477 patch_prober.go:29] interesting pod/e2e-inf-a-6922d3-openshift-default-689b5f95bc-x68sw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:42:38.500160 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:38.499809    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-x68sw" podUID="ca4b71ad-f474-4db6-a350-8f1da8fe6244" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:42:41.500424 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:41.500396    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-7bhqj"
Oct 30 22:42:41.622555 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:41.622525    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-txq8f"
Oct 30 22:42:41.627312 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:41.627278    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-x68sw"
Oct 30 22:42:41.637930 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:41.637899    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58693405-8a5b-4841-865d-aa29cc320e4d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58693405-8a5b-4841-865d-aa29cc320e4d-credential-socket\") pod \"58693405-8a5b-4841-865d-aa29cc320e4d\" (UID: \"58693405-8a5b-4841-865d-aa29cc320e4d\") "
Oct 30 22:42:41.638075 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:41.638009    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58693405-8a5b-4841-865d-aa29cc320e4d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58693405-8a5b-4841-865d-aa29cc320e4d-workload-socket\") pod \"58693405-8a5b-4841-865d-aa29cc320e4d\" (UID: \"58693405-8a5b-4841-865d-aa29cc320e4d\") "
Oct 30 22:42:41.638160 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:41.638143    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58693405-8a5b-4841-865d-aa29cc320e4d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58693405-8a5b-4841-865d-aa29cc320e4d-istio-data\") pod \"58693405-8a5b-4841-865d-aa29cc320e4d\" (UID: \"58693405-8a5b-4841-865d-aa29cc320e4d\") "
Oct 30 22:42:41.638227 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:41.638196    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58693405-8a5b-4841-865d-aa29cc320e4d-istio-token\" (UniqueName: \"kubernetes.io/projected/58693405-8a5b-4841-865d-aa29cc320e4d-istio-token\") pod \"58693405-8a5b-4841-865d-aa29cc320e4d\" (UID: \"58693405-8a5b-4841-865d-aa29cc320e4d\") "
Oct 30 22:42:41.638227 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:41.638197    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58693405-8a5b-4841-865d-aa29cc320e4d-credential-socket" pod "58693405-8a5b-4841-865d-aa29cc320e4d" (UID: "58693405-8a5b-4841-865d-aa29cc320e4d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:42:41.638336 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:41.638242    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58693405-8a5b-4841-865d-aa29cc320e4d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58693405-8a5b-4841-865d-aa29cc320e4d-workload-certs\") pod \"58693405-8a5b-4841-865d-aa29cc320e4d\" (UID: \"58693405-8a5b-4841-865d-aa29cc320e4d\") "
Oct 30 22:42:41.638336 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:41.638273    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58693405-8a5b-4841-865d-aa29cc320e4d-kube-api-access-slmtv\" (UniqueName: \"kubernetes.io/projected/58693405-8a5b-4841-865d-aa29cc320e4d-kube-api-access-slmtv\") pod \"58693405-8a5b-4841-865d-aa29cc320e4d\" (UID: \"58693405-8a5b-4841-865d-aa29cc320e4d\") "
Oct 30 22:42:41.638336 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:41.638306    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58693405-8a5b-4841-865d-aa29cc320e4d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58693405-8a5b-4841-865d-aa29cc320e4d-istio-envoy\") pod \"58693405-8a5b-4841-865d-aa29cc320e4d\" (UID: \"58693405-8a5b-4841-865d-aa29cc320e4d\") "
Oct 30 22:42:41.638495 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:41.638349    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58693405-8a5b-4841-865d-aa29cc320e4d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58693405-8a5b-4841-865d-aa29cc320e4d-istiod-ca-cert\") pod \"58693405-8a5b-4841-865d-aa29cc320e4d\" (UID: \"58693405-8a5b-4841-865d-aa29cc320e4d\") "
Oct 30 22:42:41.638495 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:41.638409    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58693405-8a5b-4841-865d-aa29cc320e4d-workload-socket" pod "58693405-8a5b-4841-865d-aa29cc320e4d" (UID: "58693405-8a5b-4841-865d-aa29cc320e4d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:42:41.638495 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:41.638431    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/58693405-8a5b-4841-865d-aa29cc320e4d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58693405-8a5b-4841-865d-aa29cc320e4d-istio-podinfo\") pod \"58693405-8a5b-4841-865d-aa29cc320e4d\" (UID: \"58693405-8a5b-4841-865d-aa29cc320e4d\") "
Oct 30 22:42:41.638655 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:41.638578    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58693405-8a5b-4841-865d-aa29cc320e4d-workload-certs" pod "58693405-8a5b-4841-865d-aa29cc320e4d" (UID: "58693405-8a5b-4841-865d-aa29cc320e4d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:42:41.638780 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:41.638759    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58693405-8a5b-4841-865d-aa29cc320e4d-istio-data" pod "58693405-8a5b-4841-865d-aa29cc320e4d" (UID: "58693405-8a5b-4841-865d-aa29cc320e4d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:42:41.639130 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:41.639111    4477 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58693405-8a5b-4841-865d-aa29cc320e4d-istio-data\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:42:41.639254 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:41.639225    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58693405-8a5b-4841-865d-aa29cc320e4d-istiod-ca-cert" pod "58693405-8a5b-4841-865d-aa29cc320e4d" (UID: "58693405-8a5b-4841-865d-aa29cc320e4d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:42:41.639351 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:41.639246    4477 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58693405-8a5b-4841-865d-aa29cc320e4d-workload-certs\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:42:41.639450 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:41.639437    4477 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58693405-8a5b-4841-865d-aa29cc320e4d-credential-socket\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:42:41.639554 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:41.639539    4477 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58693405-8a5b-4841-865d-aa29cc320e4d-workload-socket\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:42:41.641344 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:41.641321    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58693405-8a5b-4841-865d-aa29cc320e4d-istio-envoy" pod "58693405-8a5b-4841-865d-aa29cc320e4d" (UID: "58693405-8a5b-4841-865d-aa29cc320e4d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:42:41.641491 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:41.641418    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/58693405-8a5b-4841-865d-aa29cc320e4d-istio-podinfo" pod "58693405-8a5b-4841-865d-aa29cc320e4d" (UID: "58693405-8a5b-4841-865d-aa29cc320e4d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:42:41.642122 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:41.642098    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58693405-8a5b-4841-865d-aa29cc320e4d-kube-api-access-slmtv" pod "58693405-8a5b-4841-865d-aa29cc320e4d" (UID: "58693405-8a5b-4841-865d-aa29cc320e4d"). InnerVolumeSpecName "kube-api-access-slmtv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:42:41.642423 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:41.642399    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58693405-8a5b-4841-865d-aa29cc320e4d-istio-token" pod "58693405-8a5b-4841-865d-aa29cc320e4d" (UID: "58693405-8a5b-4841-865d-aa29cc320e4d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:42:41.742116 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:41.740305    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca4b71ad-f474-4db6-a350-8f1da8fe6244-istio-token\" (UniqueName: \"kubernetes.io/projected/ca4b71ad-f474-4db6-a350-8f1da8fe6244-istio-token\") pod \"ca4b71ad-f474-4db6-a350-8f1da8fe6244\" (UID: \"ca4b71ad-f474-4db6-a350-8f1da8fe6244\") "
Oct 30 22:42:41.742116 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:41.740355    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca4b71ad-f474-4db6-a350-8f1da8fe6244-kube-api-access-n6jlq\" (UniqueName: \"kubernetes.io/projected/ca4b71ad-f474-4db6-a350-8f1da8fe6244-kube-api-access-n6jlq\") pod \"ca4b71ad-f474-4db6-a350-8f1da8fe6244\" (UID: \"ca4b71ad-f474-4db6-a350-8f1da8fe6244\") "
Oct 30 22:42:41.742116 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:41.740436    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95030005-bb27-46a1-9238-503138ca97f6-istio-token\" (UniqueName: \"kubernetes.io/projected/95030005-bb27-46a1-9238-503138ca97f6-istio-token\") pod \"95030005-bb27-46a1-9238-503138ca97f6\" (UID: \"95030005-bb27-46a1-9238-503138ca97f6\") "
Oct 30 22:42:41.742116 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:41.740513    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95030005-bb27-46a1-9238-503138ca97f6-kube-api-access-wmj89\" (UniqueName: \"kubernetes.io/projected/95030005-bb27-46a1-9238-503138ca97f6-kube-api-access-wmj89\") pod \"95030005-bb27-46a1-9238-503138ca97f6\" (UID: \"95030005-bb27-46a1-9238-503138ca97f6\") "
Oct 30 22:42:41.742116 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:41.740550    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ca4b71ad-f474-4db6-a350-8f1da8fe6244-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca4b71ad-f474-4db6-a350-8f1da8fe6244-istio-podinfo\") pod \"ca4b71ad-f474-4db6-a350-8f1da8fe6244\" (UID: \"ca4b71ad-f474-4db6-a350-8f1da8fe6244\") "
Oct 30 22:42:41.742116 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:41.740592    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca4b71ad-f474-4db6-a350-8f1da8fe6244-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca4b71ad-f474-4db6-a350-8f1da8fe6244-workload-socket\") pod \"ca4b71ad-f474-4db6-a350-8f1da8fe6244\" (UID: \"ca4b71ad-f474-4db6-a350-8f1da8fe6244\") "
Oct 30 22:42:41.742116 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:41.740647    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca4b71ad-f474-4db6-a350-8f1da8fe6244-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca4b71ad-f474-4db6-a350-8f1da8fe6244-istio-envoy\") pod \"ca4b71ad-f474-4db6-a350-8f1da8fe6244\" (UID: \"ca4b71ad-f474-4db6-a350-8f1da8fe6244\") "
Oct 30 22:42:41.742116 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:41.740686    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95030005-bb27-46a1-9238-503138ca97f6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95030005-bb27-46a1-9238-503138ca97f6-workload-socket\") pod \"95030005-bb27-46a1-9238-503138ca97f6\" (UID: \"95030005-bb27-46a1-9238-503138ca97f6\") "
Oct 30 22:42:41.742116 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:41.740761    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95030005-bb27-46a1-9238-503138ca97f6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95030005-bb27-46a1-9238-503138ca97f6-istio-data\") pod \"95030005-bb27-46a1-9238-503138ca97f6\" (UID: \"95030005-bb27-46a1-9238-503138ca97f6\") "
Oct 30 22:42:41.742116 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:41.740802    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95030005-bb27-46a1-9238-503138ca97f6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95030005-bb27-46a1-9238-503138ca97f6-istio-envoy\") pod \"95030005-bb27-46a1-9238-503138ca97f6\" (UID: \"95030005-bb27-46a1-9238-503138ca97f6\") "
Oct 30 22:42:41.742116 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:41.740900    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca4b71ad-f474-4db6-a350-8f1da8fe6244-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca4b71ad-f474-4db6-a350-8f1da8fe6244-credential-socket\") pod \"ca4b71ad-f474-4db6-a350-8f1da8fe6244\" (UID: \"ca4b71ad-f474-4db6-a350-8f1da8fe6244\") "
Oct 30 22:42:41.742116 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:41.740939    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95030005-bb27-46a1-9238-503138ca97f6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95030005-bb27-46a1-9238-503138ca97f6-credential-socket\") pod \"95030005-bb27-46a1-9238-503138ca97f6\" (UID: \"95030005-bb27-46a1-9238-503138ca97f6\") "
Oct 30 22:42:41.742116 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:41.740997    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95030005-bb27-46a1-9238-503138ca97f6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95030005-bb27-46a1-9238-503138ca97f6-workload-certs\") pod \"95030005-bb27-46a1-9238-503138ca97f6\" (UID: \"95030005-bb27-46a1-9238-503138ca97f6\") "
Oct 30 22:42:41.742116 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:41.741025    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95030005-bb27-46a1-9238-503138ca97f6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95030005-bb27-46a1-9238-503138ca97f6-istiod-ca-cert\") pod \"95030005-bb27-46a1-9238-503138ca97f6\" (UID: \"95030005-bb27-46a1-9238-503138ca97f6\") "
Oct 30 22:42:41.742116 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:41.741106    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca4b71ad-f474-4db6-a350-8f1da8fe6244-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca4b71ad-f474-4db6-a350-8f1da8fe6244-istio-data\") pod \"ca4b71ad-f474-4db6-a350-8f1da8fe6244\" (UID: \"ca4b71ad-f474-4db6-a350-8f1da8fe6244\") "
Oct 30 22:42:41.742116 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:41.741141    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca4b71ad-f474-4db6-a350-8f1da8fe6244-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca4b71ad-f474-4db6-a350-8f1da8fe6244-workload-certs\") pod \"ca4b71ad-f474-4db6-a350-8f1da8fe6244\" (UID: \"ca4b71ad-f474-4db6-a350-8f1da8fe6244\") "
Oct 30 22:42:41.742116 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:41.741204    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/95030005-bb27-46a1-9238-503138ca97f6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95030005-bb27-46a1-9238-503138ca97f6-istio-podinfo\") pod \"95030005-bb27-46a1-9238-503138ca97f6\" (UID: \"95030005-bb27-46a1-9238-503138ca97f6\") "
Oct 30 22:42:41.742116 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:41.741315    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca4b71ad-f474-4db6-a350-8f1da8fe6244-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca4b71ad-f474-4db6-a350-8f1da8fe6244-istiod-ca-cert\") pod \"ca4b71ad-f474-4db6-a350-8f1da8fe6244\" (UID: \"ca4b71ad-f474-4db6-a350-8f1da8fe6244\") "
Oct 30 22:42:41.742886 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:41.742328    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95030005-bb27-46a1-9238-503138ca97f6-istio-data" pod "95030005-bb27-46a1-9238-503138ca97f6" (UID: "95030005-bb27-46a1-9238-503138ca97f6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:42:41.743386 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:41.743357    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95030005-bb27-46a1-9238-503138ca97f6-workload-certs" pod "95030005-bb27-46a1-9238-503138ca97f6" (UID: "95030005-bb27-46a1-9238-503138ca97f6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:42:41.744556 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:41.744496    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95030005-bb27-46a1-9238-503138ca97f6-istiod-ca-cert" pod "95030005-bb27-46a1-9238-503138ca97f6" (UID: "95030005-bb27-46a1-9238-503138ca97f6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:42:41.744945 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:41.744918    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca4b71ad-f474-4db6-a350-8f1da8fe6244-istio-data" pod "ca4b71ad-f474-4db6-a350-8f1da8fe6244" (UID: "ca4b71ad-f474-4db6-a350-8f1da8fe6244"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:42:41.745065 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:41.744961    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca4b71ad-f474-4db6-a350-8f1da8fe6244-credential-socket" pod "ca4b71ad-f474-4db6-a350-8f1da8fe6244" (UID: "ca4b71ad-f474-4db6-a350-8f1da8fe6244"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:42:41.745223 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:41.745201    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca4b71ad-f474-4db6-a350-8f1da8fe6244-workload-certs" pod "ca4b71ad-f474-4db6-a350-8f1da8fe6244" (UID: "ca4b71ad-f474-4db6-a350-8f1da8fe6244"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:42:41.746641 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:41.746615    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95030005-bb27-46a1-9238-503138ca97f6-credential-socket" pod "95030005-bb27-46a1-9238-503138ca97f6" (UID: "95030005-bb27-46a1-9238-503138ca97f6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:42:41.746814 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:41.746795    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca4b71ad-f474-4db6-a350-8f1da8fe6244-workload-socket" pod "ca4b71ad-f474-4db6-a350-8f1da8fe6244" (UID: "ca4b71ad-f474-4db6-a350-8f1da8fe6244"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:42:41.746967 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:41.746950    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95030005-bb27-46a1-9238-503138ca97f6-workload-socket" pod "95030005-bb27-46a1-9238-503138ca97f6" (UID: "95030005-bb27-46a1-9238-503138ca97f6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:42:41.747315 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:41.747286    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca4b71ad-f474-4db6-a350-8f1da8fe6244-istiod-ca-cert" pod "ca4b71ad-f474-4db6-a350-8f1da8fe6244" (UID: "ca4b71ad-f474-4db6-a350-8f1da8fe6244"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:42:41.747872 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:41.747847    4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slmtv\" (UniqueName: \"kubernetes.io/projected/58693405-8a5b-4841-865d-aa29cc320e4d-kube-api-access-slmtv\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:42:41.747988 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:41.747875    4477 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58693405-8a5b-4841-865d-aa29cc320e4d-istio-envoy\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:42:41.747988 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:41.747890    4477 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58693405-8a5b-4841-865d-aa29cc320e4d-istiod-ca-cert\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:42:41.747988 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:41.747903    4477 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95030005-bb27-46a1-9238-503138ca97f6-workload-socket\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:42:41.747988 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:41.747916    4477 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95030005-bb27-46a1-9238-503138ca97f6-istio-data\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:42:41.747988 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:41.747929    4477 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58693405-8a5b-4841-865d-aa29cc320e4d-istio-podinfo\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:42:41.747988 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:41.747942    4477 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca4b71ad-f474-4db6-a350-8f1da8fe6244-credential-socket\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:42:41.747988 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:41.747955    4477 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95030005-bb27-46a1-9238-503138ca97f6-credential-socket\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:42:41.747988 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:41.747968    4477 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95030005-bb27-46a1-9238-503138ca97f6-workload-certs\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:42:41.747988 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:41.747981    4477 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95030005-bb27-46a1-9238-503138ca97f6-istiod-ca-cert\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:42:41.747988 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:41.747993    4477 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca4b71ad-f474-4db6-a350-8f1da8fe6244-istio-data\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:42:41.748528 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:41.748006    4477 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca4b71ad-f474-4db6-a350-8f1da8fe6244-workload-certs\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:42:41.748528 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:41.748018    4477 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca4b71ad-f474-4db6-a350-8f1da8fe6244-istiod-ca-cert\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:42:41.748528 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:41.748030    4477 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58693405-8a5b-4841-865d-aa29cc320e4d-istio-token\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:42:41.748528 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:41.748081    4477 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca4b71ad-f474-4db6-a350-8f1da8fe6244-workload-socket\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:42:41.748826 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:41.748784    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca4b71ad-f474-4db6-a350-8f1da8fe6244-kube-api-access-n6jlq" pod "ca4b71ad-f474-4db6-a350-8f1da8fe6244" (UID: "ca4b71ad-f474-4db6-a350-8f1da8fe6244"). InnerVolumeSpecName "kube-api-access-n6jlq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:42:41.750260 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:41.750238    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/95030005-bb27-46a1-9238-503138ca97f6-istio-podinfo" pod "95030005-bb27-46a1-9238-503138ca97f6" (UID: "95030005-bb27-46a1-9238-503138ca97f6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:42:41.753488 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:41.753455    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95030005-bb27-46a1-9238-503138ca97f6-kube-api-access-wmj89" pod "95030005-bb27-46a1-9238-503138ca97f6" (UID: "95030005-bb27-46a1-9238-503138ca97f6"). InnerVolumeSpecName "kube-api-access-wmj89". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:42:41.753586 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:41.753524    4477 generic.go:356] "Generic (PLEG): container finished" podID="ca4b71ad-f474-4db6-a350-8f1da8fe6244" containerID="64627c9156678f089e05be2217551c32c7a9e7303a7f30a797dc1d24c08235b5" exitCode=0
Oct 30 22:42:41.754154 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:41.753687    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-x68sw" event={"ID":"ca4b71ad-f474-4db6-a350-8f1da8fe6244","Type":"ContainerDied","Data":"64627c9156678f089e05be2217551c32c7a9e7303a7f30a797dc1d24c08235b5"}
Oct 30 22:42:41.754154 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:41.753725    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-x68sw"
Oct 30 22:42:41.754154 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:41.753760    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-x68sw" event={"ID":"ca4b71ad-f474-4db6-a350-8f1da8fe6244","Type":"ContainerDied","Data":"ba8d08b4641c31efb55b3a07f60e1cfaa8a7def418dcd98b20aab78f1507a7db"}
Oct 30 22:42:41.754154 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:41.753795    4477 scope.go:122] "RemoveContainer" containerID="64627c9156678f089e05be2217551c32c7a9e7303a7f30a797dc1d24c08235b5"
Oct 30 22:42:41.755248 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:41.755224    4477 generic.go:356] "Generic (PLEG): container finished" podID="58693405-8a5b-4841-865d-aa29cc320e4d" containerID="27913e02cf990f9b7065601c31973768d95c7cfc49c140fb17043f00795551a1" exitCode=0
Oct 30 22:42:41.755332 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:41.755220    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95030005-bb27-46a1-9238-503138ca97f6-istio-token" pod "95030005-bb27-46a1-9238-503138ca97f6" (UID: "95030005-bb27-46a1-9238-503138ca97f6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:42:41.755332 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:41.755320    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-7bhqj" event={"ID":"58693405-8a5b-4841-865d-aa29cc320e4d","Type":"ContainerDied","Data":"27913e02cf990f9b7065601c31973768d95c7cfc49c140fb17043f00795551a1"}
Oct 30 22:42:41.755459 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:41.755348    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-7bhqj"
Oct 30 22:42:41.755459 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:41.755365    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-7bhqj" event={"ID":"58693405-8a5b-4841-865d-aa29cc320e4d","Type":"ContainerDied","Data":"1edf19ed8cd3c4cd4b9e246a49dbc154833867571d8ddf7025e5834697bc91bf"}
Oct 30 22:42:41.756227 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:41.756205    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ca4b71ad-f474-4db6-a350-8f1da8fe6244-istio-podinfo" pod "ca4b71ad-f474-4db6-a350-8f1da8fe6244" (UID: "ca4b71ad-f474-4db6-a350-8f1da8fe6244"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:42:41.765484 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:41.764961    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca4b71ad-f474-4db6-a350-8f1da8fe6244-istio-token" pod "ca4b71ad-f474-4db6-a350-8f1da8fe6244" (UID: "ca4b71ad-f474-4db6-a350-8f1da8fe6244"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:42:41.765484 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:41.765033    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95030005-bb27-46a1-9238-503138ca97f6-istio-envoy" pod "95030005-bb27-46a1-9238-503138ca97f6" (UID: "95030005-bb27-46a1-9238-503138ca97f6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:42:41.765484 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:41.765117    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca4b71ad-f474-4db6-a350-8f1da8fe6244-istio-envoy" pod "ca4b71ad-f474-4db6-a350-8f1da8fe6244" (UID: "ca4b71ad-f474-4db6-a350-8f1da8fe6244"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:42:41.769541 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:41.769513    4477 generic.go:356] "Generic (PLEG): container finished" podID="95030005-bb27-46a1-9238-503138ca97f6" containerID="22294cb010b5e5259f33a86d207c642f175cfb7afcf2d391740469eec302ee14" exitCode=0
Oct 30 22:42:41.769630 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:41.769613    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-txq8f" event={"ID":"95030005-bb27-46a1-9238-503138ca97f6","Type":"ContainerDied","Data":"22294cb010b5e5259f33a86d207c642f175cfb7afcf2d391740469eec302ee14"}
Oct 30 22:42:41.769702 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:41.769660    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-txq8f" event={"ID":"95030005-bb27-46a1-9238-503138ca97f6","Type":"ContainerDied","Data":"2d0b83ff2af69adffc5f2f7b9fcd3d7184cec909bbf86bcf199a22ab74e87818"}
Oct 30 22:42:41.769770 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:41.769733    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-txq8f"
Oct 30 22:42:41.771058 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:41.771029    4477 scope.go:122] "RemoveContainer" containerID="64627c9156678f089e05be2217551c32c7a9e7303a7f30a797dc1d24c08235b5"
Oct 30 22:42:41.771596 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:42:41.771461    4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64627c9156678f089e05be2217551c32c7a9e7303a7f30a797dc1d24c08235b5\": container with ID starting with 64627c9156678f089e05be2217551c32c7a9e7303a7f30a797dc1d24c08235b5 not found: ID does not exist" containerID="64627c9156678f089e05be2217551c32c7a9e7303a7f30a797dc1d24c08235b5"
Oct 30 22:42:41.771596 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:41.771491    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64627c9156678f089e05be2217551c32c7a9e7303a7f30a797dc1d24c08235b5"} err="failed to get container status \"64627c9156678f089e05be2217551c32c7a9e7303a7f30a797dc1d24c08235b5\": rpc error: code = NotFound desc = could not find container \"64627c9156678f089e05be2217551c32c7a9e7303a7f30a797dc1d24c08235b5\": container with ID starting with 64627c9156678f089e05be2217551c32c7a9e7303a7f30a797dc1d24c08235b5 not found: ID does not exist"
Oct 30 22:42:41.771596 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:41.771516    4477 scope.go:122] "RemoveContainer" containerID="27913e02cf990f9b7065601c31973768d95c7cfc49c140fb17043f00795551a1"
Oct 30 22:42:41.788766 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:41.788742    4477 scope.go:122] "RemoveContainer" containerID="27913e02cf990f9b7065601c31973768d95c7cfc49c140fb17043f00795551a1"
Oct 30 22:42:41.789001 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:42:41.788983    4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27913e02cf990f9b7065601c31973768d95c7cfc49c140fb17043f00795551a1\": container with ID starting with 27913e02cf990f9b7065601c31973768d95c7cfc49c140fb17043f00795551a1 not found: ID does not exist" containerID="27913e02cf990f9b7065601c31973768d95c7cfc49c140fb17043f00795551a1"
Oct 30 22:42:41.789088 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:41.789005    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27913e02cf990f9b7065601c31973768d95c7cfc49c140fb17043f00795551a1"} err="failed to get container status \"27913e02cf990f9b7065601c31973768d95c7cfc49c140fb17043f00795551a1\": rpc error: code = NotFound desc = could not find container \"27913e02cf990f9b7065601c31973768d95c7cfc49c140fb17043f00795551a1\": container with ID starting with 27913e02cf990f9b7065601c31973768d95c7cfc49c140fb17043f00795551a1 not found: ID does not exist"
Oct 30 22:42:41.789088 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:41.789020    4477 scope.go:122] "RemoveContainer" containerID="22294cb010b5e5259f33a86d207c642f175cfb7afcf2d391740469eec302ee14"
Oct 30 22:42:41.800747 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:41.800729    4477 scope.go:122] "RemoveContainer" containerID="22294cb010b5e5259f33a86d207c642f175cfb7afcf2d391740469eec302ee14"
Oct 30 22:42:41.801012 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:42:41.800996    4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22294cb010b5e5259f33a86d207c642f175cfb7afcf2d391740469eec302ee14\": container with ID starting with 22294cb010b5e5259f33a86d207c642f175cfb7afcf2d391740469eec302ee14 not found: ID does not exist" containerID="22294cb010b5e5259f33a86d207c642f175cfb7afcf2d391740469eec302ee14"
Oct 30 22:42:41.801129 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:41.801020    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22294cb010b5e5259f33a86d207c642f175cfb7afcf2d391740469eec302ee14"} err="failed to get container status \"22294cb010b5e5259f33a86d207c642f175cfb7afcf2d391740469eec302ee14\": rpc error: code = NotFound desc = could not find container \"22294cb010b5e5259f33a86d207c642f175cfb7afcf2d391740469eec302ee14\": container with ID starting with 22294cb010b5e5259f33a86d207c642f175cfb7afcf2d391740469eec302ee14 not found: ID does not exist"
Oct 30 22:42:41.825326 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:41.825300    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-txq8f"]
Oct 30 22:42:41.834144 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:41.834122    4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-txq8f"]
Oct 30 22:42:41.850293 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:41.850268    4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmj89\" (UniqueName: \"kubernetes.io/projected/95030005-bb27-46a1-9238-503138ca97f6-kube-api-access-wmj89\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:42:41.850380 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:41.850298    4477 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca4b71ad-f474-4db6-a350-8f1da8fe6244-istio-podinfo\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:42:41.850380 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:41.850313    4477 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca4b71ad-f474-4db6-a350-8f1da8fe6244-istio-envoy\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:42:41.850380 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:41.850326    4477 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95030005-bb27-46a1-9238-503138ca97f6-istio-envoy\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:42:41.850380 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:41.850339    4477 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95030005-bb27-46a1-9238-503138ca97f6-istio-podinfo\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:42:41.850380 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:41.850351    4477 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca4b71ad-f474-4db6-a350-8f1da8fe6244-istio-token\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:42:41.850380 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:41.850363    4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6jlq\" (UniqueName: \"kubernetes.io/projected/ca4b71ad-f474-4db6-a350-8f1da8fe6244-kube-api-access-n6jlq\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:42:41.850380 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:41.850375    4477 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95030005-bb27-46a1-9238-503138ca97f6-istio-token\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:42:41.866882 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:41.866864    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-7bhqj"]
Oct 30 22:42:41.870164 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:41.870143    4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-7bhqj"]
Oct 30 22:42:42.101600 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:42.101567    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-x68sw"]
Oct 30 22:42:42.113307 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:42.113272    4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-6922d3-openshift-default-689b5f95bc-x68sw"]
Oct 30 22:42:42.194570 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:42.194541    4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58693405-8a5b-4841-865d-aa29cc320e4d" path="/var/lib/kubelet/pods/58693405-8a5b-4841-865d-aa29cc320e4d/volumes"
Oct 30 22:42:42.195050 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:42.195023    4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95030005-bb27-46a1-9238-503138ca97f6" path="/var/lib/kubelet/pods/95030005-bb27-46a1-9238-503138ca97f6/volumes"
Oct 30 22:42:42.195517 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:42.195502    4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca4b71ad-f474-4db6-a350-8f1da8fe6244" path="/var/lib/kubelet/pods/ca4b71ad-f474-4db6-a350-8f1da8fe6244/volumes"
Oct 30 22:42:56.032064 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:42:56.029131    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-bdd94b/test-model-bdd94b-kserve-86fcb699c-svmrn"]
Oct 30 22:42:56.283394 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:42:56.283310    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 22:43:11.061597 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:43:11.061542    4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-bdd94b/test-model-bdd94b-kserve-86fcb699c-svmrn" podUID="6a0bd85e-66f4-4942-840a-7f91e0413a8a" containerName="main" containerID="cri-o://7bbd121e65b9d9201b3ff4d5acc00ec0160a6fd877efd16006e871192b0c9966" gracePeriod=45
Oct 30 22:43:11.091259 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:43:11.091225    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-bdd94b/test-model-bdd94b-kserve-86fcb699c-svmrn" podUID="6a0bd85e-66f4-4942-840a-7f91e0413a8a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.118:8000/ready\": dial tcp 10.134.0.118:8000: connect: connection refused"
Oct 30 22:43:11.345527 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:43:11.345494    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-bdd94b/test-model-bdd94b-kserve-86fcb699c-svmrn"
Oct 30 22:43:11.434942 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:43:11.434909    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a0bd85e-66f4-4942-840a-7f91e0413a8a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a0bd85e-66f4-4942-840a-7f91e0413a8a-tmp-dir\") pod \"6a0bd85e-66f4-4942-840a-7f91e0413a8a\" (UID: \"6a0bd85e-66f4-4942-840a-7f91e0413a8a\") "
Oct 30 22:43:11.435158 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:43:11.435137    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a0bd85e-66f4-4942-840a-7f91e0413a8a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a0bd85e-66f4-4942-840a-7f91e0413a8a-model-cache\") pod \"6a0bd85e-66f4-4942-840a-7f91e0413a8a\" (UID: \"6a0bd85e-66f4-4942-840a-7f91e0413a8a\") "
Oct 30 22:43:11.435245 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:43:11.435212    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a0bd85e-66f4-4942-840a-7f91e0413a8a-tmp-dir" pod "6a0bd85e-66f4-4942-840a-7f91e0413a8a" (UID: "6a0bd85e-66f4-4942-840a-7f91e0413a8a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:43:11.435313 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:43:11.435259    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a0bd85e-66f4-4942-840a-7f91e0413a8a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a0bd85e-66f4-4942-840a-7f91e0413a8a-dshm\") pod \"6a0bd85e-66f4-4942-840a-7f91e0413a8a\" (UID: \"6a0bd85e-66f4-4942-840a-7f91e0413a8a\") "
Oct 30 22:43:11.435373 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:43:11.435338    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a0bd85e-66f4-4942-840a-7f91e0413a8a-home\" (UniqueName: \"kubernetes.io/empty-dir/6a0bd85e-66f4-4942-840a-7f91e0413a8a-home\") pod \"6a0bd85e-66f4-4942-840a-7f91e0413a8a\" (UID: \"6a0bd85e-66f4-4942-840a-7f91e0413a8a\") "
Oct 30 22:43:11.435457 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:43:11.435430    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a0bd85e-66f4-4942-840a-7f91e0413a8a-model-cache" pod "6a0bd85e-66f4-4942-840a-7f91e0413a8a" (UID: "6a0bd85e-66f4-4942-840a-7f91e0413a8a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:43:11.435522 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:43:11.435451    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a0bd85e-66f4-4942-840a-7f91e0413a8a-kube-api-access-l9fwm\" (UniqueName: \"kubernetes.io/projected/6a0bd85e-66f4-4942-840a-7f91e0413a8a-kube-api-access-l9fwm\") pod \"6a0bd85e-66f4-4942-840a-7f91e0413a8a\" (UID: \"6a0bd85e-66f4-4942-840a-7f91e0413a8a\") "
Oct 30 22:43:11.435584 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:43:11.435540    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a0bd85e-66f4-4942-840a-7f91e0413a8a-tls-certs\" (UniqueName: \"kubernetes.io/secret/6a0bd85e-66f4-4942-840a-7f91e0413a8a-tls-certs\") pod \"6a0bd85e-66f4-4942-840a-7f91e0413a8a\" (UID: \"6a0bd85e-66f4-4942-840a-7f91e0413a8a\") "
Oct 30 22:43:11.435584 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:43:11.435570    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a0bd85e-66f4-4942-840a-7f91e0413a8a-home" pod "6a0bd85e-66f4-4942-840a-7f91e0413a8a" (UID: "6a0bd85e-66f4-4942-840a-7f91e0413a8a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:43:11.436444 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:43:11.436416    4477 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a0bd85e-66f4-4942-840a-7f91e0413a8a-home\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:43:11.436444 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:43:11.436445    4477 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a0bd85e-66f4-4942-840a-7f91e0413a8a-tmp-dir\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:43:11.436654 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:43:11.436460    4477 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a0bd85e-66f4-4942-840a-7f91e0413a8a-model-cache\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:43:11.438083 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:43:11.438036    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a0bd85e-66f4-4942-840a-7f91e0413a8a-kube-api-access-l9fwm" pod "6a0bd85e-66f4-4942-840a-7f91e0413a8a" (UID: "6a0bd85e-66f4-4942-840a-7f91e0413a8a"). InnerVolumeSpecName "kube-api-access-l9fwm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:43:11.438180 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:43:11.438137    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a0bd85e-66f4-4942-840a-7f91e0413a8a-tls-certs" pod "6a0bd85e-66f4-4942-840a-7f91e0413a8a" (UID: "6a0bd85e-66f4-4942-840a-7f91e0413a8a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:43:11.440130 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:43:11.440112    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a0bd85e-66f4-4942-840a-7f91e0413a8a-dshm" pod "6a0bd85e-66f4-4942-840a-7f91e0413a8a" (UID: "6a0bd85e-66f4-4942-840a-7f91e0413a8a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:43:11.537651 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:43:11.537617    4477 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a0bd85e-66f4-4942-840a-7f91e0413a8a-dshm\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:43:11.537651 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:43:11.537646    4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9fwm\" (UniqueName: \"kubernetes.io/projected/6a0bd85e-66f4-4942-840a-7f91e0413a8a-kube-api-access-l9fwm\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:43:11.537834 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:43:11.537660    4477 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a0bd85e-66f4-4942-840a-7f91e0413a8a-tls-certs\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:43:11.988394 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:43:11.988355    4477 generic.go:356] "Generic (PLEG): container finished" podID="6a0bd85e-66f4-4942-840a-7f91e0413a8a" containerID="7bbd121e65b9d9201b3ff4d5acc00ec0160a6fd877efd16006e871192b0c9966" exitCode=0
Oct 30 22:43:11.988588 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:43:11.988414    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-bdd94b/test-model-bdd94b-kserve-86fcb699c-svmrn"
Oct 30 22:43:11.988588 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:43:11.988414    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-bdd94b/test-model-bdd94b-kserve-86fcb699c-svmrn" event={"ID":"6a0bd85e-66f4-4942-840a-7f91e0413a8a","Type":"ContainerDied","Data":"7bbd121e65b9d9201b3ff4d5acc00ec0160a6fd877efd16006e871192b0c9966"}
Oct 30 22:43:11.988588 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:43:11.988559    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-bdd94b/test-model-bdd94b-kserve-86fcb699c-svmrn" event={"ID":"6a0bd85e-66f4-4942-840a-7f91e0413a8a","Type":"ContainerDied","Data":"e2b5e75f0c2d37d7227d4b1807225aebb2f464443b4b2dbeeee45faa01d29ec5"}
Oct 30 22:43:11.988892 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:43:11.988615    4477 scope.go:122] "RemoveContainer" containerID="7bbd121e65b9d9201b3ff4d5acc00ec0160a6fd877efd16006e871192b0c9966"
Oct 30 22:43:12.001003 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:43:12.000982    4477 scope.go:122] "RemoveContainer" containerID="7bbd121e65b9d9201b3ff4d5acc00ec0160a6fd877efd16006e871192b0c9966"
Oct 30 22:43:12.001285 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:43:12.001266    4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bbd121e65b9d9201b3ff4d5acc00ec0160a6fd877efd16006e871192b0c9966\": container with ID starting with 7bbd121e65b9d9201b3ff4d5acc00ec0160a6fd877efd16006e871192b0c9966 not found: ID does not exist" containerID="7bbd121e65b9d9201b3ff4d5acc00ec0160a6fd877efd16006e871192b0c9966"
Oct 30 22:43:12.001341 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:43:12.001293    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bbd121e65b9d9201b3ff4d5acc00ec0160a6fd877efd16006e871192b0c9966"} err="failed to get container status \"7bbd121e65b9d9201b3ff4d5acc00ec0160a6fd877efd16006e871192b0c9966\": rpc error: code = NotFound desc = could not find container \"7bbd121e65b9d9201b3ff4d5acc00ec0160a6fd877efd16006e871192b0c9966\": container with ID starting with 7bbd121e65b9d9201b3ff4d5acc00ec0160a6fd877efd16006e871192b0c9966 not found: ID does not exist"
Oct 30 22:43:12.048884 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:43:12.048855    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-bdd94b/test-model-bdd94b-kserve-86fcb699c-svmrn"]
Oct 30 22:43:12.060017 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:43:12.059993    4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-bdd94b/test-model-bdd94b-kserve-86fcb699c-svmrn"]
Oct 30 22:43:12.195363 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:43:12.195335    4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a0bd85e-66f4-4942-840a-7f91e0413a8a" path="/var/lib/kubelet/pods/6a0bd85e-66f4-4942-840a-7f91e0413a8a/volumes"
Oct 30 22:43:22.459433 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:43:22.459399    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6bz92"]
Oct 30 22:43:26.266605 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:43:26.266569    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 22:43:26.658407 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:43:26.658378    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-spjhs"]
Oct 30 22:43:26.658584 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:43:26.658545    4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-spjhs" podUID="5e1402af-909e-4800-953a-5b92138194dc" containerName="istio-proxy" containerID="cri-o://58fb6b84e0a11aeda8513e1ff24b66a60c058eb06a4ed70fbe70310dd0fbac1d" gracePeriod=30
Oct 30 22:43:31.935177 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:43:31.935120    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-spjhs"
Oct 30 22:43:32.029661 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:43:32.029633    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e1402af-909e-4800-953a-5b92138194dc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e1402af-909e-4800-953a-5b92138194dc-workload-socket\") pod \"5e1402af-909e-4800-953a-5b92138194dc\" (UID: \"5e1402af-909e-4800-953a-5b92138194dc\") "
Oct 30 22:43:32.029847 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:43:32.029674    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5e1402af-909e-4800-953a-5b92138194dc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e1402af-909e-4800-953a-5b92138194dc-istio-podinfo\") pod \"5e1402af-909e-4800-953a-5b92138194dc\" (UID: \"5e1402af-909e-4800-953a-5b92138194dc\") "
Oct 30 22:43:32.029847 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:43:32.029775    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e1402af-909e-4800-953a-5b92138194dc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e1402af-909e-4800-953a-5b92138194dc-istio-data\") pod \"5e1402af-909e-4800-953a-5b92138194dc\" (UID: \"5e1402af-909e-4800-953a-5b92138194dc\") "
Oct 30 22:43:32.029975 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:43:32.029903    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e1402af-909e-4800-953a-5b92138194dc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e1402af-909e-4800-953a-5b92138194dc-istio-envoy\") pod \"5e1402af-909e-4800-953a-5b92138194dc\" (UID: \"5e1402af-909e-4800-953a-5b92138194dc\") "
Oct 30 22:43:32.030028 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:43:32.029951    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e1402af-909e-4800-953a-5b92138194dc-workload-socket" pod "5e1402af-909e-4800-953a-5b92138194dc" (UID: "5e1402af-909e-4800-953a-5b92138194dc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:43:32.030028 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:43:32.029996    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e1402af-909e-4800-953a-5b92138194dc-istio-token\" (UniqueName: \"kubernetes.io/projected/5e1402af-909e-4800-953a-5b92138194dc-istio-token\") pod \"5e1402af-909e-4800-953a-5b92138194dc\" (UID: \"5e1402af-909e-4800-953a-5b92138194dc\") "
Oct 30 22:43:32.030183 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:43:32.030026    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e1402af-909e-4800-953a-5b92138194dc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e1402af-909e-4800-953a-5b92138194dc-workload-certs\") pod \"5e1402af-909e-4800-953a-5b92138194dc\" (UID: \"5e1402af-909e-4800-953a-5b92138194dc\") "
Oct 30 22:43:32.030183 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:43:32.030027    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e1402af-909e-4800-953a-5b92138194dc-istio-data" pod "5e1402af-909e-4800-953a-5b92138194dc" (UID: "5e1402af-909e-4800-953a-5b92138194dc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:43:32.030183 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:43:32.030099    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e1402af-909e-4800-953a-5b92138194dc-kube-api-access-8hqzk\" (UniqueName: \"kubernetes.io/projected/5e1402af-909e-4800-953a-5b92138194dc-kube-api-access-8hqzk\") pod \"5e1402af-909e-4800-953a-5b92138194dc\" (UID: \"5e1402af-909e-4800-953a-5b92138194dc\") "
Oct 30 22:43:32.030351 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:43:32.030197    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e1402af-909e-4800-953a-5b92138194dc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e1402af-909e-4800-953a-5b92138194dc-istiod-ca-cert\") pod \"5e1402af-909e-4800-953a-5b92138194dc\" (UID: \"5e1402af-909e-4800-953a-5b92138194dc\") "
Oct 30 22:43:32.030351 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:43:32.030314    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e1402af-909e-4800-953a-5b92138194dc-workload-certs" pod "5e1402af-909e-4800-953a-5b92138194dc" (UID: "5e1402af-909e-4800-953a-5b92138194dc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:43:32.030351 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:43:32.030321    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e1402af-909e-4800-953a-5b92138194dc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e1402af-909e-4800-953a-5b92138194dc-credential-socket\") pod \"5e1402af-909e-4800-953a-5b92138194dc\" (UID: \"5e1402af-909e-4800-953a-5b92138194dc\") "
Oct 30 22:43:32.030721 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:43:32.030695    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e1402af-909e-4800-953a-5b92138194dc-istiod-ca-cert" pod "5e1402af-909e-4800-953a-5b92138194dc" (UID: "5e1402af-909e-4800-953a-5b92138194dc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:43:32.030790 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:43:32.030763    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e1402af-909e-4800-953a-5b92138194dc-credential-socket" pod "5e1402af-909e-4800-953a-5b92138194dc" (UID: "5e1402af-909e-4800-953a-5b92138194dc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:43:32.031009 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:43:32.030987    4477 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e1402af-909e-4800-953a-5b92138194dc-istiod-ca-cert\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:43:32.031104 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:43:32.031014    4477 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e1402af-909e-4800-953a-5b92138194dc-credential-socket\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:43:32.031104 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:43:32.031030    4477 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e1402af-909e-4800-953a-5b92138194dc-workload-socket\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:43:32.031230 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:43:32.031155    4477 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e1402af-909e-4800-953a-5b92138194dc-istio-data\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:43:32.031230 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:43:32.031171    4477 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e1402af-909e-4800-953a-5b92138194dc-workload-certs\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:43:32.032897 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:43:32.032870    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e1402af-909e-4800-953a-5b92138194dc-kube-api-access-8hqzk" pod "5e1402af-909e-4800-953a-5b92138194dc" (UID: "5e1402af-909e-4800-953a-5b92138194dc"). InnerVolumeSpecName "kube-api-access-8hqzk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:43:32.034468 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:43:32.034441    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5e1402af-909e-4800-953a-5b92138194dc-istio-podinfo" pod "5e1402af-909e-4800-953a-5b92138194dc" (UID: "5e1402af-909e-4800-953a-5b92138194dc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:43:32.034557 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:43:32.034476    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e1402af-909e-4800-953a-5b92138194dc-istio-envoy" pod "5e1402af-909e-4800-953a-5b92138194dc" (UID: "5e1402af-909e-4800-953a-5b92138194dc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:43:32.035444 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:43:32.035424    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e1402af-909e-4800-953a-5b92138194dc-istio-token" pod "5e1402af-909e-4800-953a-5b92138194dc" (UID: "5e1402af-909e-4800-953a-5b92138194dc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:43:32.133092 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:43:32.133064    4477 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e1402af-909e-4800-953a-5b92138194dc-istio-envoy\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:43:32.133092 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:43:32.133089    4477 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e1402af-909e-4800-953a-5b92138194dc-istio-token\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:43:32.133092 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:43:32.133098    4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hqzk\" (UniqueName: \"kubernetes.io/projected/5e1402af-909e-4800-953a-5b92138194dc-kube-api-access-8hqzk\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:43:32.133320 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:43:32.133107    4477 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e1402af-909e-4800-953a-5b92138194dc-istio-podinfo\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:43:32.148654 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:43:32.148611    4477 generic.go:356] "Generic (PLEG): container finished" podID="5e1402af-909e-4800-953a-5b92138194dc" containerID="58fb6b84e0a11aeda8513e1ff24b66a60c058eb06a4ed70fbe70310dd0fbac1d" exitCode=0
Oct 30 22:43:32.148804 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:43:32.148668    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-spjhs"
Oct 30 22:43:32.148804 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:43:32.148670    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-spjhs" event={"ID":"5e1402af-909e-4800-953a-5b92138194dc","Type":"ContainerDied","Data":"58fb6b84e0a11aeda8513e1ff24b66a60c058eb06a4ed70fbe70310dd0fbac1d"}
Oct 30 22:43:32.148909 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:43:32.148806    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-spjhs" event={"ID":"5e1402af-909e-4800-953a-5b92138194dc","Type":"ContainerDied","Data":"e1a3ce4334135ac88c367ba88b23c025e085a16194ae3ea403b127cd5b97f910"}
Oct 30 22:43:32.148909 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:43:32.148830    4477 scope.go:122] "RemoveContainer" containerID="58fb6b84e0a11aeda8513e1ff24b66a60c058eb06a4ed70fbe70310dd0fbac1d"
Oct 30 22:43:32.164071 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:43:32.161010    4477 scope.go:122] "RemoveContainer" containerID="58fb6b84e0a11aeda8513e1ff24b66a60c058eb06a4ed70fbe70310dd0fbac1d"
Oct 30 22:43:32.164071 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:43:32.163296    4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58fb6b84e0a11aeda8513e1ff24b66a60c058eb06a4ed70fbe70310dd0fbac1d\": container with ID starting with 58fb6b84e0a11aeda8513e1ff24b66a60c058eb06a4ed70fbe70310dd0fbac1d not found: ID does not exist" containerID="58fb6b84e0a11aeda8513e1ff24b66a60c058eb06a4ed70fbe70310dd0fbac1d"
Oct 30 22:43:32.164071 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:43:32.163332    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58fb6b84e0a11aeda8513e1ff24b66a60c058eb06a4ed70fbe70310dd0fbac1d"} err="failed to get container status \"58fb6b84e0a11aeda8513e1ff24b66a60c058eb06a4ed70fbe70310dd0fbac1d\": rpc error: code = NotFound desc = could not find container \"58fb6b84e0a11aeda8513e1ff24b66a60c058eb06a4ed70fbe70310dd0fbac1d\": container with ID starting with 58fb6b84e0a11aeda8513e1ff24b66a60c058eb06a4ed70fbe70310dd0fbac1d not found: ID does not exist"
Oct 30 22:43:32.201998 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:43:32.201936    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-spjhs"]
Oct 30 22:43:32.205811 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:43:32.205785    4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-421583-openshift-default-5cb767b447-spjhs"]
Oct 30 22:43:34.193889 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:43:34.193857    4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e1402af-909e-4800-953a-5b92138194dc" path="/var/lib/kubelet/pods/5e1402af-909e-4800-953a-5b92138194dc/volumes"
Oct 30 22:43:56.268755 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:43:56.268681    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 22:44:25.641991 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:25.641952    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-frdpl"]
Oct 30 22:44:25.642572 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:25.642214    4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-frdpl" podUID="d3b4a8a3-deec-47d0-9692-4f44a319f922" containerName="istio-proxy" containerID="cri-o://67263e866672e1e481a94bc3b912551703ef7626d36390da30239e3c097f1de2" gracePeriod=30
Oct 30 22:44:25.651566 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:25.651535    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hv7z4"]
Oct 30 22:44:25.651722 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:25.651699    4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hv7z4" podUID="8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d" containerName="istio-proxy" containerID="cri-o://f35c87e53b11d3dcb55acab8d5a6a8c385a6c29e5972da230e3011295ba5cbb8" gracePeriod=30
Oct 30 22:44:26.269714 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:44:26.269678    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 22:44:30.929033 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:30.928972    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-frdpl"
Oct 30 22:44:30.973923 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:30.973903    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hv7z4"
Oct 30 22:44:31.090259 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:31.090231    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d-istio-envoy\") pod \"8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d\" (UID: \"8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d\") "
Oct 30 22:44:31.090443 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:31.090295    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d3b4a8a3-deec-47d0-9692-4f44a319f922-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3b4a8a3-deec-47d0-9692-4f44a319f922-istio-podinfo\") pod \"d3b4a8a3-deec-47d0-9692-4f44a319f922\" (UID: \"d3b4a8a3-deec-47d0-9692-4f44a319f922\") "
Oct 30 22:44:31.090443 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:31.090345    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d-istiod-ca-cert\") pod \"8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d\" (UID: \"8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d\") "
Oct 30 22:44:31.090443 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:31.090367    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d-istio-data\") pod \"8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d\" (UID: \"8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d\") "
Oct 30 22:44:31.090443 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:31.090417    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3b4a8a3-deec-47d0-9692-4f44a319f922-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3b4a8a3-deec-47d0-9692-4f44a319f922-istiod-ca-cert\") pod \"d3b4a8a3-deec-47d0-9692-4f44a319f922\" (UID: \"d3b4a8a3-deec-47d0-9692-4f44a319f922\") "
Oct 30 22:44:31.090686 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:31.090447    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d-credential-socket\") pod \"8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d\" (UID: \"8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d\") "
Oct 30 22:44:31.090686 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:31.090502    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3b4a8a3-deec-47d0-9692-4f44a319f922-istio-token\" (UniqueName: \"kubernetes.io/projected/d3b4a8a3-deec-47d0-9692-4f44a319f922-istio-token\") pod \"d3b4a8a3-deec-47d0-9692-4f44a319f922\" (UID: \"d3b4a8a3-deec-47d0-9692-4f44a319f922\") "
Oct 30 22:44:31.090686 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:31.090539    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d-kube-api-access-5d2fv\" (UniqueName: \"kubernetes.io/projected/8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d-kube-api-access-5d2fv\") pod \"8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d\" (UID: \"8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d\") "
Oct 30 22:44:31.090686 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:31.090572    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d-istio-token\" (UniqueName: \"kubernetes.io/projected/8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d-istio-token\") pod \"8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d\" (UID: \"8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d\") "
Oct 30 22:44:31.090686 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:31.090625    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3b4a8a3-deec-47d0-9692-4f44a319f922-kube-api-access-87jb5\" (UniqueName: \"kubernetes.io/projected/d3b4a8a3-deec-47d0-9692-4f44a319f922-kube-api-access-87jb5\") pod \"d3b4a8a3-deec-47d0-9692-4f44a319f922\" (UID: \"d3b4a8a3-deec-47d0-9692-4f44a319f922\") "
Oct 30 22:44:31.090956 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:31.090696    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3b4a8a3-deec-47d0-9692-4f44a319f922-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3b4a8a3-deec-47d0-9692-4f44a319f922-credential-socket\") pod \"d3b4a8a3-deec-47d0-9692-4f44a319f922\" (UID: \"d3b4a8a3-deec-47d0-9692-4f44a319f922\") "
Oct 30 22:44:31.090956 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:31.090731    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3b4a8a3-deec-47d0-9692-4f44a319f922-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3b4a8a3-deec-47d0-9692-4f44a319f922-istio-envoy\") pod \"d3b4a8a3-deec-47d0-9692-4f44a319f922\" (UID: \"d3b4a8a3-deec-47d0-9692-4f44a319f922\") "
Oct 30 22:44:31.090956 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:31.090694    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d-istio-data" pod "8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d" (UID: "8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:44:31.090956 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:31.090761    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3b4a8a3-deec-47d0-9692-4f44a319f922-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3b4a8a3-deec-47d0-9692-4f44a319f922-workload-certs\") pod \"d3b4a8a3-deec-47d0-9692-4f44a319f922\" (UID: \"d3b4a8a3-deec-47d0-9692-4f44a319f922\") "
Oct 30 22:44:31.090956 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:31.090782    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d-credential-socket" pod "8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d" (UID: "8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:44:31.090956 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:31.090797    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d-istiod-ca-cert" pod "8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d" (UID: "8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:44:31.090956 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:31.090835    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3b4a8a3-deec-47d0-9692-4f44a319f922-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3b4a8a3-deec-47d0-9692-4f44a319f922-istio-data\") pod \"d3b4a8a3-deec-47d0-9692-4f44a319f922\" (UID: \"d3b4a8a3-deec-47d0-9692-4f44a319f922\") "
Oct 30 22:44:31.090956 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:31.090861    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3b4a8a3-deec-47d0-9692-4f44a319f922-istiod-ca-cert" pod "d3b4a8a3-deec-47d0-9692-4f44a319f922" (UID: "d3b4a8a3-deec-47d0-9692-4f44a319f922"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:44:31.090956 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:31.090897    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d-workload-certs\") pod \"8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d\" (UID: \"8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d\") "
Oct 30 22:44:31.091473 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:31.090974    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d-workload-socket\") pod \"8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d\" (UID: \"8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d\") "
Oct 30 22:44:31.091473 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:31.091007    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d-istio-podinfo\") pod \"8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d\" (UID: \"8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d\") "
Oct 30 22:44:31.091473 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:31.091117    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3b4a8a3-deec-47d0-9692-4f44a319f922-credential-socket" pod "d3b4a8a3-deec-47d0-9692-4f44a319f922" (UID: "d3b4a8a3-deec-47d0-9692-4f44a319f922"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:44:31.091473 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:31.091177    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3b4a8a3-deec-47d0-9692-4f44a319f922-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3b4a8a3-deec-47d0-9692-4f44a319f922-workload-socket\") pod \"d3b4a8a3-deec-47d0-9692-4f44a319f922\" (UID: \"d3b4a8a3-deec-47d0-9692-4f44a319f922\") "
Oct 30 22:44:31.091473 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:31.091339    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3b4a8a3-deec-47d0-9692-4f44a319f922-istio-data" pod "d3b4a8a3-deec-47d0-9692-4f44a319f922" (UID: "d3b4a8a3-deec-47d0-9692-4f44a319f922"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:44:31.091733 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:31.091655    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d-workload-socket" pod "8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d" (UID: "8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:44:31.091894 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:31.091872    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3b4a8a3-deec-47d0-9692-4f44a319f922-workload-socket" pod "d3b4a8a3-deec-47d0-9692-4f44a319f922" (UID: "d3b4a8a3-deec-47d0-9692-4f44a319f922"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:44:31.091995 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:31.091913    4477 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3b4a8a3-deec-47d0-9692-4f44a319f922-credential-socket\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:44:31.091995 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:31.091932    4477 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3b4a8a3-deec-47d0-9692-4f44a319f922-istio-data\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:44:31.091995 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:31.091947    4477 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d-workload-socket\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:44:31.091995 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:31.091959    4477 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d-istiod-ca-cert\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:44:31.091995 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:31.091972    4477 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d-istio-data\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:44:31.091995 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:31.091984    4477 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3b4a8a3-deec-47d0-9692-4f44a319f922-istiod-ca-cert\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:44:31.092353 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:31.091996    4477 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d-credential-socket\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:44:31.092353 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:31.092068    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d-workload-certs" pod "8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d" (UID: "8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:44:31.092353 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:31.092075    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3b4a8a3-deec-47d0-9692-4f44a319f922-workload-certs" pod "d3b4a8a3-deec-47d0-9692-4f44a319f922" (UID: "d3b4a8a3-deec-47d0-9692-4f44a319f922"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:44:31.093110 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:31.093085    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d3b4a8a3-deec-47d0-9692-4f44a319f922-istio-podinfo" pod "d3b4a8a3-deec-47d0-9692-4f44a319f922" (UID: "d3b4a8a3-deec-47d0-9692-4f44a319f922"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:44:31.093499 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:31.093437    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3b4a8a3-deec-47d0-9692-4f44a319f922-istio-token" pod "d3b4a8a3-deec-47d0-9692-4f44a319f922" (UID: "d3b4a8a3-deec-47d0-9692-4f44a319f922"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:44:31.093499 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:31.093464    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3b4a8a3-deec-47d0-9692-4f44a319f922-kube-api-access-87jb5" pod "d3b4a8a3-deec-47d0-9692-4f44a319f922" (UID: "d3b4a8a3-deec-47d0-9692-4f44a319f922"). InnerVolumeSpecName "kube-api-access-87jb5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:44:31.093683 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:31.093534    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3b4a8a3-deec-47d0-9692-4f44a319f922-istio-envoy" pod "d3b4a8a3-deec-47d0-9692-4f44a319f922" (UID: "d3b4a8a3-deec-47d0-9692-4f44a319f922"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:44:31.093942 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:31.093922    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d-kube-api-access-5d2fv" pod "8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d" (UID: "8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d"). InnerVolumeSpecName "kube-api-access-5d2fv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:44:31.094007 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:31.093992    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d-istio-podinfo" pod "8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d" (UID: "8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:44:31.094799 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:31.094766    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d-istio-envoy" pod "8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d" (UID: "8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:44:31.094891 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:31.094861    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d-istio-token" pod "8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d" (UID: "8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:44:31.193235 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:31.193171    4477 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3b4a8a3-deec-47d0-9692-4f44a319f922-istio-token\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:44:31.193235 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:31.193193    4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5d2fv\" (UniqueName: \"kubernetes.io/projected/8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d-kube-api-access-5d2fv\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:44:31.193235 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:31.193202    4477 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d-istio-token\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:44:31.193235 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:31.193211    4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87jb5\" (UniqueName: \"kubernetes.io/projected/d3b4a8a3-deec-47d0-9692-4f44a319f922-kube-api-access-87jb5\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:44:31.193235 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:31.193219    4477 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3b4a8a3-deec-47d0-9692-4f44a319f922-istio-envoy\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:44:31.193235 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:31.193227    4477 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3b4a8a3-deec-47d0-9692-4f44a319f922-workload-certs\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:44:31.193235 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:31.193234    4477 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d-workload-certs\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:44:31.193235 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:31.193242    4477 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d-istio-podinfo\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:44:31.193566 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:31.193249    4477 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3b4a8a3-deec-47d0-9692-4f44a319f922-workload-socket\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:44:31.193566 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:31.193257    4477 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d-istio-envoy\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:44:31.193566 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:31.193266    4477 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3b4a8a3-deec-47d0-9692-4f44a319f922-istio-podinfo\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:44:31.566201 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:31.566117    4477 generic.go:356] "Generic (PLEG): container finished" podID="d3b4a8a3-deec-47d0-9692-4f44a319f922" containerID="67263e866672e1e481a94bc3b912551703ef7626d36390da30239e3c097f1de2" exitCode=0
Oct 30 22:44:31.566201 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:31.566164    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-frdpl" event={"ID":"d3b4a8a3-deec-47d0-9692-4f44a319f922","Type":"ContainerDied","Data":"67263e866672e1e481a94bc3b912551703ef7626d36390da30239e3c097f1de2"}
Oct 30 22:44:31.566424 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:31.566256    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-frdpl" event={"ID":"d3b4a8a3-deec-47d0-9692-4f44a319f922","Type":"ContainerDied","Data":"8415c50c98f4464565b26d369188e21a1149f86558b34d402cb007f4cbc7d941"}
Oct 30 22:44:31.566424 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:31.566192    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-frdpl"
Oct 30 22:44:31.566424 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:31.566282    4477 scope.go:122] "RemoveContainer" containerID="67263e866672e1e481a94bc3b912551703ef7626d36390da30239e3c097f1de2"
Oct 30 22:44:31.568132 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:31.568103    4477 generic.go:356] "Generic (PLEG): container finished" podID="8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d" containerID="f35c87e53b11d3dcb55acab8d5a6a8c385a6c29e5972da230e3011295ba5cbb8" exitCode=0
Oct 30 22:44:31.568223 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:31.568202    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hv7z4" event={"ID":"8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d","Type":"ContainerDied","Data":"f35c87e53b11d3dcb55acab8d5a6a8c385a6c29e5972da230e3011295ba5cbb8"}
Oct 30 22:44:31.568286 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:31.568252    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hv7z4" event={"ID":"8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d","Type":"ContainerDied","Data":"561c6af02147db61debe1c956b38b8c66083cf009784484265105d748428bae5"}
Oct 30 22:44:31.568438 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:31.568411    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hv7z4"
Oct 30 22:44:31.579511 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:31.579491    4477 scope.go:122] "RemoveContainer" containerID="67263e866672e1e481a94bc3b912551703ef7626d36390da30239e3c097f1de2"
Oct 30 22:44:31.579801 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:44:31.579779    4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67263e866672e1e481a94bc3b912551703ef7626d36390da30239e3c097f1de2\": container with ID starting with 67263e866672e1e481a94bc3b912551703ef7626d36390da30239e3c097f1de2 not found: ID does not exist" containerID="67263e866672e1e481a94bc3b912551703ef7626d36390da30239e3c097f1de2"
Oct 30 22:44:31.579892 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:31.579810    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67263e866672e1e481a94bc3b912551703ef7626d36390da30239e3c097f1de2"} err="failed to get container status \"67263e866672e1e481a94bc3b912551703ef7626d36390da30239e3c097f1de2\": rpc error: code = NotFound desc = could not find container \"67263e866672e1e481a94bc3b912551703ef7626d36390da30239e3c097f1de2\": container with ID starting with 67263e866672e1e481a94bc3b912551703ef7626d36390da30239e3c097f1de2 not found: ID does not exist"
Oct 30 22:44:31.579892 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:31.579828    4477 scope.go:122] "RemoveContainer" containerID="f35c87e53b11d3dcb55acab8d5a6a8c385a6c29e5972da230e3011295ba5cbb8"
Oct 30 22:44:31.589317 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:31.589294    4477 scope.go:122] "RemoveContainer" containerID="f35c87e53b11d3dcb55acab8d5a6a8c385a6c29e5972da230e3011295ba5cbb8"
Oct 30 22:44:31.589613 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:44:31.589590    4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f35c87e53b11d3dcb55acab8d5a6a8c385a6c29e5972da230e3011295ba5cbb8\": container with ID starting with f35c87e53b11d3dcb55acab8d5a6a8c385a6c29e5972da230e3011295ba5cbb8 not found: ID does not exist" containerID="f35c87e53b11d3dcb55acab8d5a6a8c385a6c29e5972da230e3011295ba5cbb8"
Oct 30 22:44:31.589692 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:31.589623    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f35c87e53b11d3dcb55acab8d5a6a8c385a6c29e5972da230e3011295ba5cbb8"} err="failed to get container status \"f35c87e53b11d3dcb55acab8d5a6a8c385a6c29e5972da230e3011295ba5cbb8\": rpc error: code = NotFound desc = could not find container \"f35c87e53b11d3dcb55acab8d5a6a8c385a6c29e5972da230e3011295ba5cbb8\": container with ID starting with f35c87e53b11d3dcb55acab8d5a6a8c385a6c29e5972da230e3011295ba5cbb8 not found: ID does not exist"
Oct 30 22:44:31.632017 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:31.631987    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hv7z4"]
Oct 30 22:44:31.643463 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:31.643437    4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-hv7z4"]
Oct 30 22:44:31.693104 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:31.693020    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-frdpl"]
Oct 30 22:44:31.696159 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:31.696083    4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-frdpl"]
Oct 30 22:44:32.195498 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:32.195466    4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d" path="/var/lib/kubelet/pods/8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d/volumes"
Oct 30 22:44:32.195935 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:32.195921    4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3b4a8a3-deec-47d0-9692-4f44a319f922" path="/var/lib/kubelet/pods/d3b4a8a3-deec-47d0-9692-4f44a319f922/volumes"
Oct 30 22:44:54.983322 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:54.983291    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-q7w2l"]
Oct 30 22:44:54.983927 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:54.983912    4477 cpu_manager.go:414] "containerMap: removing container" podUID="6a0bd85e-66f4-4942-840a-7f91e0413a8a" containerName="main"
Oct 30 22:44:54.983967 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:54.983928    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a0bd85e-66f4-4942-840a-7f91e0413a8a" containerName="main"
Oct 30 22:44:54.983967 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:54.983952    4477 cpu_manager.go:414] "containerMap: removing container" podUID="4e931bc1-b014-48ed-beaa-e82a64000d6a" containerName="maas-api"
Oct 30 22:44:54.983967 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:54.983958    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e931bc1-b014-48ed-beaa-e82a64000d6a" containerName="maas-api"
Oct 30 22:44:54.984127 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:54.983968    4477 cpu_manager.go:414] "containerMap: removing container" podUID="8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d" containerName="istio-proxy"
Oct 30 22:44:54.984127 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:54.983973    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d" containerName="istio-proxy"
Oct 30 22:44:54.984127 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:54.983980    4477 cpu_manager.go:414] "containerMap: removing container" podUID="58693405-8a5b-4841-865d-aa29cc320e4d" containerName="istio-proxy"
Oct 30 22:44:54.984127 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:54.983984    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58693405-8a5b-4841-865d-aa29cc320e4d" containerName="istio-proxy"
Oct 30 22:44:54.984127 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:54.984001    4477 cpu_manager.go:414] "containerMap: removing container" podUID="5e1402af-909e-4800-953a-5b92138194dc" containerName="istio-proxy"
Oct 30 22:44:54.984127 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:54.984006    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e1402af-909e-4800-953a-5b92138194dc" containerName="istio-proxy"
Oct 30 22:44:54.984127 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:54.984014    4477 cpu_manager.go:414] "containerMap: removing container" podUID="95030005-bb27-46a1-9238-503138ca97f6" containerName="istio-proxy"
Oct 30 22:44:54.984127 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:54.984018    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95030005-bb27-46a1-9238-503138ca97f6" containerName="istio-proxy"
Oct 30 22:44:54.984127 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:54.984026    4477 cpu_manager.go:414] "containerMap: removing container" podUID="cccc800e-68e3-40ff-b7a0-29f5a777e83b" containerName="main"
Oct 30 22:44:54.984127 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:54.984030    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cccc800e-68e3-40ff-b7a0-29f5a777e83b" containerName="main"
Oct 30 22:44:54.984127 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:54.984057    4477 cpu_manager.go:414] "containerMap: removing container" podUID="0a99ab39-789a-4cbf-8db5-5e85bfe485c2" containerName="istio-proxy"
Oct 30 22:44:54.984127 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:54.984062    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a99ab39-789a-4cbf-8db5-5e85bfe485c2" containerName="istio-proxy"
Oct 30 22:44:54.984127 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:54.984070    4477 cpu_manager.go:414] "containerMap: removing container" podUID="d3b4a8a3-deec-47d0-9692-4f44a319f922" containerName="istio-proxy"
Oct 30 22:44:54.984127 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:54.984074    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3b4a8a3-deec-47d0-9692-4f44a319f922" containerName="istio-proxy"
Oct 30 22:44:54.984127 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:54.984080    4477 cpu_manager.go:414] "containerMap: removing container" podUID="5e7cf3ef-0cd6-415f-8fb5-b806fd818c77" containerName="main"
Oct 30 22:44:54.984127 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:54.984084    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e7cf3ef-0cd6-415f-8fb5-b806fd818c77" containerName="main"
Oct 30 22:44:54.984127 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:54.984097    4477 cpu_manager.go:414] "containerMap: removing container" podUID="ca4b71ad-f474-4db6-a350-8f1da8fe6244" containerName="istio-proxy"
Oct 30 22:44:54.984127 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:54.984102    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca4b71ad-f474-4db6-a350-8f1da8fe6244" containerName="istio-proxy"
Oct 30 22:44:54.984680 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:54.984194    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca4b71ad-f474-4db6-a350-8f1da8fe6244" containerName="istio-proxy"
Oct 30 22:44:54.984680 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:54.984204    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a0bd85e-66f4-4942-840a-7f91e0413a8a" containerName="main"
Oct 30 22:44:54.984680 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:54.984213    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e931bc1-b014-48ed-beaa-e82a64000d6a" containerName="maas-api"
Oct 30 22:44:54.984680 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:54.984219    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e51f3ec-6ceb-4136-9f56-eb0c2be81c7d" containerName="istio-proxy"
Oct 30 22:44:54.984680 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:54.984223    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="58693405-8a5b-4841-865d-aa29cc320e4d" containerName="istio-proxy"
Oct 30 22:44:54.984680 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:54.984230    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e1402af-909e-4800-953a-5b92138194dc" containerName="istio-proxy"
Oct 30 22:44:54.984680 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:54.984236    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="95030005-bb27-46a1-9238-503138ca97f6" containerName="istio-proxy"
Oct 30 22:44:54.984680 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:54.984243    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="cccc800e-68e3-40ff-b7a0-29f5a777e83b" containerName="main"
Oct 30 22:44:54.984680 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:54.984249    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a99ab39-789a-4cbf-8db5-5e85bfe485c2" containerName="istio-proxy"
Oct 30 22:44:54.984680 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:54.984255    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3b4a8a3-deec-47d0-9692-4f44a319f922" containerName="istio-proxy"
Oct 30 22:44:54.984680 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:54.984260    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e7cf3ef-0cd6-415f-8fb5-b806fd818c77" containerName="main"
Oct 30 22:44:54.987714 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:54.987696    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-q7w2l"
Oct 30 22:44:55.003183 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:55.003158    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-6e0edc-openshift-default-dockercfg-pkdjc\""
Oct 30 22:44:55.019090 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:55.019068    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-q7w2l"]
Oct 30 22:44:55.105173 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:55.105136    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39eb3b4c-a5b3-4274-86d0-335fb7851d4a-credential-socket\") pod \"e2e-inf-a-6e0edc-openshift-default-7968b4dddf-q7w2l\" (UID: \"39eb3b4c-a5b3-4274-86d0-335fb7851d4a\") " pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-q7w2l"
Oct 30 22:44:55.105340 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:55.105278    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39eb3b4c-a5b3-4274-86d0-335fb7851d4a-istio-envoy\") pod \"e2e-inf-a-6e0edc-openshift-default-7968b4dddf-q7w2l\" (UID: \"39eb3b4c-a5b3-4274-86d0-335fb7851d4a\") " pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-q7w2l"
Oct 30 22:44:55.105426 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:55.105335    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39eb3b4c-a5b3-4274-86d0-335fb7851d4a-istiod-ca-cert\") pod \"e2e-inf-a-6e0edc-openshift-default-7968b4dddf-q7w2l\" (UID: \"39eb3b4c-a5b3-4274-86d0-335fb7851d4a\") " pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-q7w2l"
Oct 30 22:44:55.105426 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:55.105405    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39eb3b4c-a5b3-4274-86d0-335fb7851d4a-workload-certs\") pod \"e2e-inf-a-6e0edc-openshift-default-7968b4dddf-q7w2l\" (UID: \"39eb3b4c-a5b3-4274-86d0-335fb7851d4a\") " pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-q7w2l"
Oct 30 22:44:55.105544 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:55.105434    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39eb3b4c-a5b3-4274-86d0-335fb7851d4a-istio-token\") pod \"e2e-inf-a-6e0edc-openshift-default-7968b4dddf-q7w2l\" (UID: \"39eb3b4c-a5b3-4274-86d0-335fb7851d4a\") " pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-q7w2l"
Oct 30 22:44:55.105544 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:55.105530    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-645tk\" (UniqueName: \"kubernetes.io/projected/39eb3b4c-a5b3-4274-86d0-335fb7851d4a-kube-api-access-645tk\") pod \"e2e-inf-a-6e0edc-openshift-default-7968b4dddf-q7w2l\" (UID: \"39eb3b4c-a5b3-4274-86d0-335fb7851d4a\") " pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-q7w2l"
Oct 30 22:44:55.105649 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:55.105590    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39eb3b4c-a5b3-4274-86d0-335fb7851d4a-istio-podinfo\") pod \"e2e-inf-a-6e0edc-openshift-default-7968b4dddf-q7w2l\" (UID: \"39eb3b4c-a5b3-4274-86d0-335fb7851d4a\") " pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-q7w2l"
Oct 30 22:44:55.105707 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:55.105654    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39eb3b4c-a5b3-4274-86d0-335fb7851d4a-istio-data\") pod \"e2e-inf-a-6e0edc-openshift-default-7968b4dddf-q7w2l\" (UID: \"39eb3b4c-a5b3-4274-86d0-335fb7851d4a\") " pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-q7w2l"
Oct 30 22:44:55.105763 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:55.105731    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39eb3b4c-a5b3-4274-86d0-335fb7851d4a-workload-socket\") pod \"e2e-inf-a-6e0edc-openshift-default-7968b4dddf-q7w2l\" (UID: \"39eb3b4c-a5b3-4274-86d0-335fb7851d4a\") " pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-q7w2l"
Oct 30 22:44:55.208619 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:55.208575    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39eb3b4c-a5b3-4274-86d0-335fb7851d4a-workload-socket\") pod \"e2e-inf-a-6e0edc-openshift-default-7968b4dddf-q7w2l\" (UID: \"39eb3b4c-a5b3-4274-86d0-335fb7851d4a\") " pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-q7w2l"
Oct 30 22:44:55.208916 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:55.208896    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39eb3b4c-a5b3-4274-86d0-335fb7851d4a-credential-socket\") pod \"e2e-inf-a-6e0edc-openshift-default-7968b4dddf-q7w2l\" (UID: \"39eb3b4c-a5b3-4274-86d0-335fb7851d4a\") " pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-q7w2l"
Oct 30 22:44:55.209110 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:55.209092    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39eb3b4c-a5b3-4274-86d0-335fb7851d4a-istio-envoy\") pod \"e2e-inf-a-6e0edc-openshift-default-7968b4dddf-q7w2l\" (UID: \"39eb3b4c-a5b3-4274-86d0-335fb7851d4a\") " pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-q7w2l"
Oct 30 22:44:55.209246 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:55.209232    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39eb3b4c-a5b3-4274-86d0-335fb7851d4a-istiod-ca-cert\") pod \"e2e-inf-a-6e0edc-openshift-default-7968b4dddf-q7w2l\" (UID: \"39eb3b4c-a5b3-4274-86d0-335fb7851d4a\") " pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-q7w2l"
Oct 30 22:44:55.209394 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:55.209379    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39eb3b4c-a5b3-4274-86d0-335fb7851d4a-workload-certs\") pod \"e2e-inf-a-6e0edc-openshift-default-7968b4dddf-q7w2l\" (UID: \"39eb3b4c-a5b3-4274-86d0-335fb7851d4a\") " pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-q7w2l"
Oct 30 22:44:55.209498 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:55.209484    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39eb3b4c-a5b3-4274-86d0-335fb7851d4a-istio-token\") pod \"e2e-inf-a-6e0edc-openshift-default-7968b4dddf-q7w2l\" (UID: \"39eb3b4c-a5b3-4274-86d0-335fb7851d4a\") " pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-q7w2l"
Oct 30 22:44:55.232232 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:55.232202    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-645tk\" (UniqueName: \"kubernetes.io/projected/39eb3b4c-a5b3-4274-86d0-335fb7851d4a-kube-api-access-645tk\") pod \"e2e-inf-a-6e0edc-openshift-default-7968b4dddf-q7w2l\" (UID: \"39eb3b4c-a5b3-4274-86d0-335fb7851d4a\") " pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-q7w2l"
Oct 30 22:44:55.232455 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:55.232439    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39eb3b4c-a5b3-4274-86d0-335fb7851d4a-istio-podinfo\") pod \"e2e-inf-a-6e0edc-openshift-default-7968b4dddf-q7w2l\" (UID: \"39eb3b4c-a5b3-4274-86d0-335fb7851d4a\") " pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-q7w2l"
Oct 30 22:44:55.246367 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:55.246300    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39eb3b4c-a5b3-4274-86d0-335fb7851d4a-istio-data\") pod \"e2e-inf-a-6e0edc-openshift-default-7968b4dddf-q7w2l\" (UID: \"39eb3b4c-a5b3-4274-86d0-335fb7851d4a\") " pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-q7w2l"
Oct 30 22:44:55.246367 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:55.214408    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39eb3b4c-a5b3-4274-86d0-335fb7851d4a-workload-certs\") pod \"e2e-inf-a-6e0edc-openshift-default-7968b4dddf-q7w2l\" (UID: \"39eb3b4c-a5b3-4274-86d0-335fb7851d4a\") " pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-q7w2l"
Oct 30 22:44:55.246548 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:55.224350    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39eb3b4c-a5b3-4274-86d0-335fb7851d4a-istio-envoy\") pod \"e2e-inf-a-6e0edc-openshift-default-7968b4dddf-q7w2l\" (UID: \"39eb3b4c-a5b3-4274-86d0-335fb7851d4a\") " pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-q7w2l"
Oct 30 22:44:55.246548 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:55.214169    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39eb3b4c-a5b3-4274-86d0-335fb7851d4a-workload-socket\") pod \"e2e-inf-a-6e0edc-openshift-default-7968b4dddf-q7w2l\" (UID: \"39eb3b4c-a5b3-4274-86d0-335fb7851d4a\") " pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-q7w2l"
Oct 30 22:44:55.246548 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:55.213850    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39eb3b4c-a5b3-4274-86d0-335fb7851d4a-istiod-ca-cert\") pod \"e2e-inf-a-6e0edc-openshift-default-7968b4dddf-q7w2l\" (UID: \"39eb3b4c-a5b3-4274-86d0-335fb7851d4a\") " pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-q7w2l"
Oct 30 22:44:55.246548 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:55.237737    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39eb3b4c-a5b3-4274-86d0-335fb7851d4a-istio-podinfo\") pod \"e2e-inf-a-6e0edc-openshift-default-7968b4dddf-q7w2l\" (UID: \"39eb3b4c-a5b3-4274-86d0-335fb7851d4a\") " pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-q7w2l"
Oct 30 22:44:55.246548 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:55.243487    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39eb3b4c-a5b3-4274-86d0-335fb7851d4a-istio-token\") pod \"e2e-inf-a-6e0edc-openshift-default-7968b4dddf-q7w2l\" (UID: \"39eb3b4c-a5b3-4274-86d0-335fb7851d4a\") " pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-q7w2l"
Oct 30 22:44:55.246548 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:55.214617    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39eb3b4c-a5b3-4274-86d0-335fb7851d4a-credential-socket\") pod \"e2e-inf-a-6e0edc-openshift-default-7968b4dddf-q7w2l\" (UID: \"39eb3b4c-a5b3-4274-86d0-335fb7851d4a\") " pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-q7w2l"
Oct 30 22:44:55.246827 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:55.246733    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39eb3b4c-a5b3-4274-86d0-335fb7851d4a-istio-data\") pod \"e2e-inf-a-6e0edc-openshift-default-7968b4dddf-q7w2l\" (UID: \"39eb3b4c-a5b3-4274-86d0-335fb7851d4a\") " pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-q7w2l"
Oct 30 22:44:55.259727 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:55.259695    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-645tk\" (UniqueName: \"kubernetes.io/projected/39eb3b4c-a5b3-4274-86d0-335fb7851d4a-kube-api-access-645tk\") pod \"e2e-inf-a-6e0edc-openshift-default-7968b4dddf-q7w2l\" (UID: \"39eb3b4c-a5b3-4274-86d0-335fb7851d4a\") " pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-q7w2l"
Oct 30 22:44:55.302365 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:55.302330    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-q7w2l"
Oct 30 22:44:55.647097 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:55.647066    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-q7w2l"]
Oct 30 22:44:55.650174 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:44:55.650150    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39eb3b4c_a5b3_4274_86d0_335fb7851d4a.slice/crio-19eba94abb4aecc70c1bf1c86c5927c9ffc80fda4ff159ae13f25319d5fcff67 WatchSource:0}: Error finding container 19eba94abb4aecc70c1bf1c86c5927c9ffc80fda4ff159ae13f25319d5fcff67: Status 404 returned error can't find the container with id 19eba94abb4aecc70c1bf1c86c5927c9ffc80fda4ff159ae13f25319d5fcff67
Oct 30 22:44:55.652795 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:55.652761    4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:44:55.652889 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:55.652829    4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:44:55.652889 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:55.652863    4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:44:55.745718 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:55.745661    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-q7w2l" event={"ID":"39eb3b4c-a5b3-4274-86d0-335fb7851d4a","Type":"ContainerStarted","Data":"47040e2bf8771f24efbf41faedee8e3d4693887b207bce53f01dc36107c13db2"}
Oct 30 22:44:55.745905 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:55.745737    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-q7w2l" event={"ID":"39eb3b4c-a5b3-4274-86d0-335fb7851d4a","Type":"ContainerStarted","Data":"19eba94abb4aecc70c1bf1c86c5927c9ffc80fda4ff159ae13f25319d5fcff67"}
Oct 30 22:44:55.791490 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:55.791424    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-q7w2l" podStartSLOduration=1.791410146 podStartE2EDuration="1.791410146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:44:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:44:55.78992268 +0000 UTC m=+2279.715848923" watchObservedRunningTime="2026-10-30 22:44:55.791410146 +0000 UTC m=+2279.717336346"
Oct 30 22:44:56.280305 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:44:56.280260    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 22:44:56.307280 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:56.307241    4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-q7w2l"
Oct 30 22:44:56.308700 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:56.308669    4477 patch_prober.go:29] interesting pod/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-q7w2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body=
Oct 30 22:44:56.308838 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:56.308726    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-q7w2l" podUID="39eb3b4c-a5b3-4274-86d0-335fb7851d4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused"
Oct 30 22:44:57.305445 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:57.305402    4477 patch_prober.go:29] interesting pod/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-q7w2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body=
Oct 30 22:44:57.305927 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:57.305465    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-q7w2l" podUID="39eb3b4c-a5b3-4274-86d0-335fb7851d4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused"
Oct 30 22:44:58.303705 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:58.303665    4477 patch_prober.go:29] interesting pod/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-q7w2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body=
Oct 30 22:44:58.303894 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:58.303731    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-q7w2l" podUID="39eb3b4c-a5b3-4274-86d0-335fb7851d4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused"
Oct 30 22:44:59.303802 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:59.303765    4477 patch_prober.go:29] interesting pod/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-q7w2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body=
Oct 30 22:44:59.304250 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:44:59.303831    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-q7w2l" podUID="39eb3b4c-a5b3-4274-86d0-335fb7851d4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused"
Oct 30 22:45:00.319323 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:45:00.319282    4477 patch_prober.go:29] interesting pod/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-q7w2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body=
Oct 30 22:45:00.319852 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:45:00.319354    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-q7w2l" podUID="39eb3b4c-a5b3-4274-86d0-335fb7851d4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused"
Oct 30 22:45:01.303580 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:45:01.303541    4477 patch_prober.go:29] interesting pod/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-q7w2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body=
Oct 30 22:45:01.303745 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:45:01.303610    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-q7w2l" podUID="39eb3b4c-a5b3-4274-86d0-335fb7851d4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused"
Oct 30 22:45:02.307670 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:45:02.307639    4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-q7w2l"
Oct 30 22:45:02.308163 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:45:02.307904    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-q7w2l"
Oct 30 22:45:02.308772 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:45:02.308747    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-q7w2l"
Oct 30 22:45:24.533499 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:45:24.533466    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6bz92"]
Oct 30 22:45:26.267718 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:45:26.267682    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 22:45:56.266881 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:45:56.266842    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 22:46:00.594862 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:46:00.594822    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-999470/test-model-999470-kserve-56bbd9b675-5xsxl"]
Oct 30 22:46:15.648217 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:46:15.648158    4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-999470/test-model-999470-kserve-56bbd9b675-5xsxl" podUID="3f3b16db-55fe-4505-9f9f-13bbfe486000" containerName="main" containerID="cri-o://6482876f76933143cb8378c3777073b4ba5244c0f5b595aa7e1565ba79eccd4c" gracePeriod=45
Oct 30 22:46:15.930133 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:46:15.930108    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-999470/test-model-999470-kserve-56bbd9b675-5xsxl"
Oct 30 22:46:16.040805 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:46:16.040774    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f3b16db-55fe-4505-9f9f-13bbfe486000-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f3b16db-55fe-4505-9f9f-13bbfe486000-model-cache\") pod \"3f3b16db-55fe-4505-9f9f-13bbfe486000\" (UID: \"3f3b16db-55fe-4505-9f9f-13bbfe486000\") "
Oct 30 22:46:16.040986 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:46:16.040890    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f3b16db-55fe-4505-9f9f-13bbfe486000-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f3b16db-55fe-4505-9f9f-13bbfe486000-dshm\") pod \"3f3b16db-55fe-4505-9f9f-13bbfe486000\" (UID: \"3f3b16db-55fe-4505-9f9f-13bbfe486000\") "
Oct 30 22:46:16.040986 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:46:16.040931    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f3b16db-55fe-4505-9f9f-13bbfe486000-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f3b16db-55fe-4505-9f9f-13bbfe486000-tmp-dir\") pod \"3f3b16db-55fe-4505-9f9f-13bbfe486000\" (UID: \"3f3b16db-55fe-4505-9f9f-13bbfe486000\") "
Oct 30 22:46:16.041157 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:46:16.040988    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f3b16db-55fe-4505-9f9f-13bbfe486000-kube-api-access-bjfsx\" (UniqueName: \"kubernetes.io/projected/3f3b16db-55fe-4505-9f9f-13bbfe486000-kube-api-access-bjfsx\") pod \"3f3b16db-55fe-4505-9f9f-13bbfe486000\" (UID: \"3f3b16db-55fe-4505-9f9f-13bbfe486000\") "
Oct 30 22:46:16.041157 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:46:16.041027    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f3b16db-55fe-4505-9f9f-13bbfe486000-model-cache" pod "3f3b16db-55fe-4505-9f9f-13bbfe486000" (UID: "3f3b16db-55fe-4505-9f9f-13bbfe486000"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:46:16.041157 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:46:16.041085    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f3b16db-55fe-4505-9f9f-13bbfe486000-home\" (UniqueName: \"kubernetes.io/empty-dir/3f3b16db-55fe-4505-9f9f-13bbfe486000-home\") pod \"3f3b16db-55fe-4505-9f9f-13bbfe486000\" (UID: \"3f3b16db-55fe-4505-9f9f-13bbfe486000\") "
Oct 30 22:46:16.041326 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:46:16.041232    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f3b16db-55fe-4505-9f9f-13bbfe486000-tls-certs\" (UniqueName: \"kubernetes.io/secret/3f3b16db-55fe-4505-9f9f-13bbfe486000-tls-certs\") pod \"3f3b16db-55fe-4505-9f9f-13bbfe486000\" (UID: \"3f3b16db-55fe-4505-9f9f-13bbfe486000\") "
Oct 30 22:46:16.041326 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:46:16.041295    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f3b16db-55fe-4505-9f9f-13bbfe486000-tmp-dir" pod "3f3b16db-55fe-4505-9f9f-13bbfe486000" (UID: "3f3b16db-55fe-4505-9f9f-13bbfe486000"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:46:16.041412 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:46:16.041337    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f3b16db-55fe-4505-9f9f-13bbfe486000-home" pod "3f3b16db-55fe-4505-9f9f-13bbfe486000" (UID: "3f3b16db-55fe-4505-9f9f-13bbfe486000"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:46:16.041837 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:46:16.041812    4477 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f3b16db-55fe-4505-9f9f-13bbfe486000-home\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:46:16.041956 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:46:16.041840    4477 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f3b16db-55fe-4505-9f9f-13bbfe486000-model-cache\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:46:16.041956 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:46:16.041858    4477 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f3b16db-55fe-4505-9f9f-13bbfe486000-tmp-dir\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:46:16.043291 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:46:16.043265    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f3b16db-55fe-4505-9f9f-13bbfe486000-dshm" pod "3f3b16db-55fe-4505-9f9f-13bbfe486000" (UID: "3f3b16db-55fe-4505-9f9f-13bbfe486000"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:46:16.043397 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:46:16.043351    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f3b16db-55fe-4505-9f9f-13bbfe486000-tls-certs" pod "3f3b16db-55fe-4505-9f9f-13bbfe486000" (UID: "3f3b16db-55fe-4505-9f9f-13bbfe486000"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:46:16.043397 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:46:16.043367    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f3b16db-55fe-4505-9f9f-13bbfe486000-kube-api-access-bjfsx" pod "3f3b16db-55fe-4505-9f9f-13bbfe486000" (UID: "3f3b16db-55fe-4505-9f9f-13bbfe486000"). InnerVolumeSpecName "kube-api-access-bjfsx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:46:16.143529 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:46:16.143504    4477 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f3b16db-55fe-4505-9f9f-13bbfe486000-tls-certs\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:46:16.143529 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:46:16.143528    4477 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f3b16db-55fe-4505-9f9f-13bbfe486000-dshm\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:46:16.143664 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:46:16.143537    4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjfsx\" (UniqueName: \"kubernetes.io/projected/3f3b16db-55fe-4505-9f9f-13bbfe486000-kube-api-access-bjfsx\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:46:16.329685 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:46:16.329654    4477 generic.go:356] "Generic (PLEG): container finished" podID="3f3b16db-55fe-4505-9f9f-13bbfe486000" containerID="6482876f76933143cb8378c3777073b4ba5244c0f5b595aa7e1565ba79eccd4c" exitCode=0
Oct 30 22:46:16.329806 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:46:16.329693    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-999470/test-model-999470-kserve-56bbd9b675-5xsxl"
Oct 30 22:46:16.329806 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:46:16.329738    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-999470/test-model-999470-kserve-56bbd9b675-5xsxl" event={"ID":"3f3b16db-55fe-4505-9f9f-13bbfe486000","Type":"ContainerDied","Data":"6482876f76933143cb8378c3777073b4ba5244c0f5b595aa7e1565ba79eccd4c"}
Oct 30 22:46:16.329806 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:46:16.329784    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-999470/test-model-999470-kserve-56bbd9b675-5xsxl" event={"ID":"3f3b16db-55fe-4505-9f9f-13bbfe486000","Type":"ContainerDied","Data":"31d155fa6a70dbc1481d5f57008ef9ba6034ef1a50c22d33253fd32df44fbbeb"}
Oct 30 22:46:16.329919 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:46:16.329805    4477 scope.go:122] "RemoveContainer" containerID="6482876f76933143cb8378c3777073b4ba5244c0f5b595aa7e1565ba79eccd4c"
Oct 30 22:46:16.339856 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:46:16.339835    4477 scope.go:122] "RemoveContainer" containerID="6482876f76933143cb8378c3777073b4ba5244c0f5b595aa7e1565ba79eccd4c"
Oct 30 22:46:16.340223 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:46:16.340200    4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6482876f76933143cb8378c3777073b4ba5244c0f5b595aa7e1565ba79eccd4c\": container with ID starting with 6482876f76933143cb8378c3777073b4ba5244c0f5b595aa7e1565ba79eccd4c not found: ID does not exist" containerID="6482876f76933143cb8378c3777073b4ba5244c0f5b595aa7e1565ba79eccd4c"
Oct 30 22:46:16.340300 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:46:16.340227    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6482876f76933143cb8378c3777073b4ba5244c0f5b595aa7e1565ba79eccd4c"} err="failed to get container status \"6482876f76933143cb8378c3777073b4ba5244c0f5b595aa7e1565ba79eccd4c\": rpc error: code = NotFound desc = could not find container \"6482876f76933143cb8378c3777073b4ba5244c0f5b595aa7e1565ba79eccd4c\": container with ID starting with 6482876f76933143cb8378c3777073b4ba5244c0f5b595aa7e1565ba79eccd4c not found: ID does not exist"
Oct 30 22:46:16.380261 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:46:16.380232    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-999470/test-model-999470-kserve-56bbd9b675-5xsxl"]
Oct 30 22:46:16.391394 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:46:16.391373    4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-999470/test-model-999470-kserve-56bbd9b675-5xsxl"]
Oct 30 22:46:18.194963 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:46:18.194930    4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f3b16db-55fe-4505-9f9f-13bbfe486000" path="/var/lib/kubelet/pods/3f3b16db-55fe-4505-9f9f-13bbfe486000/volumes"
Oct 30 22:46:26.267534 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:46:26.267495    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 22:46:56.269838 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:46:56.269795    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 22:46:56.647340 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:46:56.647312    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6bz92"]
Oct 30 22:46:56.959434 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:46:56.959366    4477 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jcf8l_015f218d-bf1f-4e7e-bb5d-b805af003246/ovn-acl-logging/0.log"
Oct 30 22:46:57.000001 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:46:56.999970    4477 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jcf8l_015f218d-bf1f-4e7e-bb5d-b805af003246/ovn-acl-logging/0.log"
Oct 30 22:47:07.670240 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:47:07.670201    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-999470-5dcc669f68-nhc2b"]
Oct 30 22:47:07.670766 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:47:07.670445    4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-999470-5dcc669f68-nhc2b" podUID="09adb84c-9a9c-466e-8e9e-8b53437593af" containerName="maas-api" containerID="cri-o://b9cff0ec35bcac86e7d049db5988b0c54ee1184ad7f906bf6cd54cb5eae4208c" gracePeriod=30
Oct 30 22:47:08.046157 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:47:08.046132    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-999470-5dcc669f68-nhc2b"
Oct 30 22:47:08.111859 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:47:08.110582    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09adb84c-9a9c-466e-8e9e-8b53437593af-metrics-certs\" (UniqueName: \"kubernetes.io/secret/09adb84c-9a9c-466e-8e9e-8b53437593af-metrics-certs\") pod \"09adb84c-9a9c-466e-8e9e-8b53437593af\" (UID: \"09adb84c-9a9c-466e-8e9e-8b53437593af\") "
Oct 30 22:47:08.111859 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:47:08.110678    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09adb84c-9a9c-466e-8e9e-8b53437593af-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/09adb84c-9a9c-466e-8e9e-8b53437593af-openshift-service-ca\") pod \"09adb84c-9a9c-466e-8e9e-8b53437593af\" (UID: \"09adb84c-9a9c-466e-8e9e-8b53437593af\") "
Oct 30 22:47:08.111859 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:47:08.110737    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09adb84c-9a9c-466e-8e9e-8b53437593af-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/09adb84c-9a9c-466e-8e9e-8b53437593af-maas-api-tls\") pod \"09adb84c-9a9c-466e-8e9e-8b53437593af\" (UID: \"09adb84c-9a9c-466e-8e9e-8b53437593af\") "
Oct 30 22:47:08.111859 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:47:08.110825    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09adb84c-9a9c-466e-8e9e-8b53437593af-kube-api-access-4jgfg\" (UniqueName: \"kubernetes.io/projected/09adb84c-9a9c-466e-8e9e-8b53437593af-kube-api-access-4jgfg\") pod \"09adb84c-9a9c-466e-8e9e-8b53437593af\" (UID: \"09adb84c-9a9c-466e-8e9e-8b53437593af\") "
Oct 30 22:47:08.112838 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:47:08.112807    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09adb84c-9a9c-466e-8e9e-8b53437593af-openshift-service-ca" pod "09adb84c-9a9c-466e-8e9e-8b53437593af" (UID: "09adb84c-9a9c-466e-8e9e-8b53437593af"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:47:08.135100 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:47:08.123338    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09adb84c-9a9c-466e-8e9e-8b53437593af-kube-api-access-4jgfg" pod "09adb84c-9a9c-466e-8e9e-8b53437593af" (UID: "09adb84c-9a9c-466e-8e9e-8b53437593af"). InnerVolumeSpecName "kube-api-access-4jgfg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:47:08.135100 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:47:08.134560    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09adb84c-9a9c-466e-8e9e-8b53437593af-maas-api-tls" pod "09adb84c-9a9c-466e-8e9e-8b53437593af" (UID: "09adb84c-9a9c-466e-8e9e-8b53437593af"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:47:08.137299 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:47:08.137258    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09adb84c-9a9c-466e-8e9e-8b53437593af-metrics-certs" pod "09adb84c-9a9c-466e-8e9e-8b53437593af" (UID: "09adb84c-9a9c-466e-8e9e-8b53437593af"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:47:08.214777 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:47:08.214738    4477 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/09adb84c-9a9c-466e-8e9e-8b53437593af-metrics-certs\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:47:08.214777 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:47:08.214778    4477 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/09adb84c-9a9c-466e-8e9e-8b53437593af-openshift-service-ca\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:47:08.215078 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:47:08.214794    4477 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/09adb84c-9a9c-466e-8e9e-8b53437593af-maas-api-tls\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:47:08.215078 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:47:08.214808    4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jgfg\" (UniqueName: \"kubernetes.io/projected/09adb84c-9a9c-466e-8e9e-8b53437593af-kube-api-access-4jgfg\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:47:08.726739 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:47:08.726704    4477 generic.go:356] "Generic (PLEG): container finished" podID="09adb84c-9a9c-466e-8e9e-8b53437593af" containerID="b9cff0ec35bcac86e7d049db5988b0c54ee1184ad7f906bf6cd54cb5eae4208c" exitCode=0
Oct 30 22:47:08.727206 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:47:08.726753    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-999470-5dcc669f68-nhc2b"
Oct 30 22:47:08.727206 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:47:08.726784    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-999470-5dcc669f68-nhc2b" event={"ID":"09adb84c-9a9c-466e-8e9e-8b53437593af","Type":"ContainerDied","Data":"b9cff0ec35bcac86e7d049db5988b0c54ee1184ad7f906bf6cd54cb5eae4208c"}
Oct 30 22:47:08.727206 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:47:08.726839    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-999470-5dcc669f68-nhc2b" event={"ID":"09adb84c-9a9c-466e-8e9e-8b53437593af","Type":"ContainerDied","Data":"18e208e6113b9d0e62b84b833fec64838eb27bab154c962f15d2f72d17d4d12d"}
Oct 30 22:47:08.727206 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:47:08.726861    4477 scope.go:122] "RemoveContainer" containerID="b9cff0ec35bcac86e7d049db5988b0c54ee1184ad7f906bf6cd54cb5eae4208c"
Oct 30 22:47:08.737506 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:47:08.737478    4477 scope.go:122] "RemoveContainer" containerID="b9cff0ec35bcac86e7d049db5988b0c54ee1184ad7f906bf6cd54cb5eae4208c"
Oct 30 22:47:08.737749 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:47:08.737732    4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9cff0ec35bcac86e7d049db5988b0c54ee1184ad7f906bf6cd54cb5eae4208c\": container with ID starting with b9cff0ec35bcac86e7d049db5988b0c54ee1184ad7f906bf6cd54cb5eae4208c not found: ID does not exist" containerID="b9cff0ec35bcac86e7d049db5988b0c54ee1184ad7f906bf6cd54cb5eae4208c"
Oct 30 22:47:08.737804 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:47:08.737759    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9cff0ec35bcac86e7d049db5988b0c54ee1184ad7f906bf6cd54cb5eae4208c"} err="failed to get container status \"b9cff0ec35bcac86e7d049db5988b0c54ee1184ad7f906bf6cd54cb5eae4208c\": rpc error: code = NotFound desc = could not find container \"b9cff0ec35bcac86e7d049db5988b0c54ee1184ad7f906bf6cd54cb5eae4208c\": container with ID starting with b9cff0ec35bcac86e7d049db5988b0c54ee1184ad7f906bf6cd54cb5eae4208c not found: ID does not exist"
Oct 30 22:47:08.778178 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:47:08.778155    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-999470-5dcc669f68-nhc2b"]
Oct 30 22:47:08.781373 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:47:08.781353    4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-999470-5dcc669f68-nhc2b"]
Oct 30 22:47:10.194567 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:47:10.194535    4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09adb84c-9a9c-466e-8e9e-8b53437593af" path="/var/lib/kubelet/pods/09adb84c-9a9c-466e-8e9e-8b53437593af/volumes"
Oct 30 22:47:11.585205 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:47:11.585169    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-znd9z"]
Oct 30 22:47:11.586154 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:47:11.586133    4477 cpu_manager.go:414] "containerMap: removing container" podUID="09adb84c-9a9c-466e-8e9e-8b53437593af" containerName="maas-api"
Oct 30 22:47:11.586242 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:47:11.586155    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09adb84c-9a9c-466e-8e9e-8b53437593af" containerName="maas-api"
Oct 30 22:47:11.586242 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:47:11.586194    4477 cpu_manager.go:414] "containerMap: removing container" podUID="3f3b16db-55fe-4505-9f9f-13bbfe486000" containerName="main"
Oct 30 22:47:11.586242 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:47:11.586202    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f3b16db-55fe-4505-9f9f-13bbfe486000" containerName="main"
Oct 30 22:47:11.586402 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:47:11.586389    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f3b16db-55fe-4505-9f9f-13bbfe486000" containerName="main"
Oct 30 22:47:11.586460 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:47:11.586416    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="09adb84c-9a9c-466e-8e9e-8b53437593af" containerName="maas-api"
Oct 30 22:47:11.592422 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:47:11.592398    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-znd9z"
Oct 30 22:47:11.606842 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:47:11.606820    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-867f12-openshift-default-dockercfg-tfg6j\""
Oct 30 22:47:11.623339 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:47:11.623316    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-znd9z"]
Oct 30 22:47:11.656274 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:47:11.656236    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2281431c-56b4-475a-a7dc-35c1bbe04e1a-istiod-ca-cert\") pod \"e2e-inf-b-867f12-openshift-default-6584f84965-znd9z\" (UID: \"2281431c-56b4-475a-a7dc-35c1bbe04e1a\") " pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-znd9z"
Oct 30 22:47:11.656421 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:47:11.656389    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwqnl\" (UniqueName: \"kubernetes.io/projected/2281431c-56b4-475a-a7dc-35c1bbe04e1a-kube-api-access-dwqnl\") pod \"e2e-inf-b-867f12-openshift-default-6584f84965-znd9z\" (UID: \"2281431c-56b4-475a-a7dc-35c1bbe04e1a\") " pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-znd9z"
Oct 30 22:47:11.656494 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:47:11.656422    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2281431c-56b4-475a-a7dc-35c1bbe04e1a-istio-data\") pod \"e2e-inf-b-867f12-openshift-default-6584f84965-znd9z\" (UID: \"2281431c-56b4-475a-a7dc-35c1bbe04e1a\") " pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-znd9z"
Oct 30 22:47:11.656587 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:47:11.656566    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2281431c-56b4-475a-a7dc-35c1bbe04e1a-workload-socket\") pod \"e2e-inf-b-867f12-openshift-default-6584f84965-znd9z\" (UID: \"2281431c-56b4-475a-a7dc-35c1bbe04e1a\") " pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-znd9z"
Oct 30 22:47:11.656657 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:47:11.656634    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2281431c-56b4-475a-a7dc-35c1bbe04e1a-istio-token\") pod \"e2e-inf-b-867f12-openshift-default-6584f84965-znd9z\" (UID: \"2281431c-56b4-475a-a7dc-35c1bbe04e1a\") " pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-znd9z"
Oct 30 22:47:11.656738 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:47:11.656722    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2281431c-56b4-475a-a7dc-35c1bbe04e1a-istio-podinfo\") pod \"e2e-inf-b-867f12-openshift-default-6584f84965-znd9z\" (UID: \"2281431c-56b4-475a-a7dc-35c1bbe04e1a\") " pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-znd9z"
Oct 30 22:47:11.656803 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:47:11.656760    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2281431c-56b4-475a-a7dc-35c1bbe04e1a-istio-envoy\") pod \"e2e-inf-b-867f12-openshift-default-6584f84965-znd9z\" (UID: \"2281431c-56b4-475a-a7dc-35c1bbe04e1a\") " pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-znd9z"
Oct 30 22:47:11.656803 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:47:11.656783    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2281431c-56b4-475a-a7dc-35c1bbe04e1a-credential-socket\") pod \"e2e-inf-b-867f12-openshift-default-6584f84965-znd9z\" (UID: \"2281431c-56b4-475a-a7dc-35c1bbe04e1a\") " pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-znd9z"
Oct 30 22:47:11.656904 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:47:11.656871    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2281431c-56b4-475a-a7dc-35c1bbe04e1a-workload-certs\") pod \"e2e-inf-b-867f12-openshift-default-6584f84965-znd9z\" (UID: \"2281431c-56b4-475a-a7dc-35c1bbe04e1a\") " pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-znd9z"
Oct 30 22:47:11.757654 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:47:11.757625    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2281431c-56b4-475a-a7dc-35c1bbe04e1a-istio-data\") pod \"e2e-inf-b-867f12-openshift-default-6584f84965-znd9z\" (UID: \"2281431c-56b4-475a-a7dc-35c1bbe04e1a\") " pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-znd9z"
Oct 30 22:47:11.757800 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:47:11.757707    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2281431c-56b4-475a-a7dc-35c1bbe04e1a-workload-socket\") pod \"e2e-inf-b-867f12-openshift-default-6584f84965-znd9z\" (UID: \"2281431c-56b4-475a-a7dc-35c1bbe04e1a\") " pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-znd9z"
Oct 30 22:47:11.757800 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:47:11.757747    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2281431c-56b4-475a-a7dc-35c1bbe04e1a-istio-token\") pod \"e2e-inf-b-867f12-openshift-default-6584f84965-znd9z\" (UID: \"2281431c-56b4-475a-a7dc-35c1bbe04e1a\") " pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-znd9z"
Oct 30 22:47:11.757883 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:47:11.757827    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2281431c-56b4-475a-a7dc-35c1bbe04e1a-istio-podinfo\") pod \"e2e-inf-b-867f12-openshift-default-6584f84965-znd9z\" (UID: \"2281431c-56b4-475a-a7dc-35c1bbe04e1a\") " pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-znd9z"
Oct 30 22:47:11.757883 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:47:11.757868    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2281431c-56b4-475a-a7dc-35c1bbe04e1a-istio-envoy\") pod \"e2e-inf-b-867f12-openshift-default-6584f84965-znd9z\" (UID: \"2281431c-56b4-475a-a7dc-35c1bbe04e1a\") " pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-znd9z"
Oct 30 22:47:11.757970 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:47:11.757902    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2281431c-56b4-475a-a7dc-35c1bbe04e1a-credential-socket\") pod \"e2e-inf-b-867f12-openshift-default-6584f84965-znd9z\" (UID: \"2281431c-56b4-475a-a7dc-35c1bbe04e1a\") " pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-znd9z"
Oct 30 22:47:11.757970 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:47:11.757936    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2281431c-56b4-475a-a7dc-35c1bbe04e1a-workload-certs\") pod \"e2e-inf-b-867f12-openshift-default-6584f84965-znd9z\" (UID: \"2281431c-56b4-475a-a7dc-35c1bbe04e1a\") " pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-znd9z"
Oct 30 22:47:11.758170 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:47:11.757979    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2281431c-56b4-475a-a7dc-35c1bbe04e1a-istiod-ca-cert\") pod \"e2e-inf-b-867f12-openshift-default-6584f84965-znd9z\" (UID: \"2281431c-56b4-475a-a7dc-35c1bbe04e1a\") " pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-znd9z"
Oct 30 22:47:11.758170 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:47:11.758013    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2281431c-56b4-475a-a7dc-35c1bbe04e1a-istio-data\") pod \"e2e-inf-b-867f12-openshift-default-6584f84965-znd9z\" (UID: \"2281431c-56b4-475a-a7dc-35c1bbe04e1a\") " pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-znd9z"
Oct 30 22:47:11.758170 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:47:11.758126    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2281431c-56b4-475a-a7dc-35c1bbe04e1a-workload-socket\") pod \"e2e-inf-b-867f12-openshift-default-6584f84965-znd9z\" (UID: \"2281431c-56b4-475a-a7dc-35c1bbe04e1a\") " pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-znd9z"
Oct 30 22:47:11.758170 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:47:11.758153    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwqnl\" (UniqueName: \"kubernetes.io/projected/2281431c-56b4-475a-a7dc-35c1bbe04e1a-kube-api-access-dwqnl\") pod \"e2e-inf-b-867f12-openshift-default-6584f84965-znd9z\" (UID: \"2281431c-56b4-475a-a7dc-35c1bbe04e1a\") " pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-znd9z"
Oct 30 22:47:11.758395 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:47:11.758341    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2281431c-56b4-475a-a7dc-35c1bbe04e1a-credential-socket\") pod \"e2e-inf-b-867f12-openshift-default-6584f84965-znd9z\" (UID: \"2281431c-56b4-475a-a7dc-35c1bbe04e1a\") " pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-znd9z"
Oct 30 22:47:11.758467 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:47:11.758411    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2281431c-56b4-475a-a7dc-35c1bbe04e1a-workload-certs\") pod \"e2e-inf-b-867f12-openshift-default-6584f84965-znd9z\" (UID: \"2281431c-56b4-475a-a7dc-35c1bbe04e1a\") " pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-znd9z"
Oct 30 22:47:11.758718 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:47:11.758701    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2281431c-56b4-475a-a7dc-35c1bbe04e1a-istiod-ca-cert\") pod \"e2e-inf-b-867f12-openshift-default-6584f84965-znd9z\" (UID: \"2281431c-56b4-475a-a7dc-35c1bbe04e1a\") " pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-znd9z"
Oct 30 22:47:11.760273 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:47:11.760249    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2281431c-56b4-475a-a7dc-35c1bbe04e1a-istio-envoy\") pod \"e2e-inf-b-867f12-openshift-default-6584f84965-znd9z\" (UID: \"2281431c-56b4-475a-a7dc-35c1bbe04e1a\") " pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-znd9z"
Oct 30 22:47:11.760421 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:47:11.760403    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2281431c-56b4-475a-a7dc-35c1bbe04e1a-istio-podinfo\") pod \"e2e-inf-b-867f12-openshift-default-6584f84965-znd9z\" (UID: \"2281431c-56b4-475a-a7dc-35c1bbe04e1a\") " pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-znd9z"
Oct 30 22:47:11.776242 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:47:11.776220    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2281431c-56b4-475a-a7dc-35c1bbe04e1a-istio-token\") pod \"e2e-inf-b-867f12-openshift-default-6584f84965-znd9z\" (UID: \"2281431c-56b4-475a-a7dc-35c1bbe04e1a\") " pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-znd9z"
Oct 30 22:47:11.776314 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:47:11.776258    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwqnl\" (UniqueName: \"kubernetes.io/projected/2281431c-56b4-475a-a7dc-35c1bbe04e1a-kube-api-access-dwqnl\") pod \"e2e-inf-b-867f12-openshift-default-6584f84965-znd9z\" (UID: \"2281431c-56b4-475a-a7dc-35c1bbe04e1a\") " pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-znd9z"
Oct 30 22:47:11.906634 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:47:11.906604    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-znd9z"
Oct 30 22:47:12.221936 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:47:12.221909    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-znd9z"]
Oct 30 22:47:12.223278 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:47:12.223250    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2281431c_56b4_475a_a7dc_35c1bbe04e1a.slice/crio-28322b7823ca3b653837b0ef429897879a0aa171ea1a3f2ce72c97ed04e8df23 WatchSource:0}: Error finding container 28322b7823ca3b653837b0ef429897879a0aa171ea1a3f2ce72c97ed04e8df23: Status 404 returned error can't find the container with id 28322b7823ca3b653837b0ef429897879a0aa171ea1a3f2ce72c97ed04e8df23
Oct 30 22:47:12.225885 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:47:12.225854    4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:47:12.225955 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:47:12.225921    4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:47:12.225955 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:47:12.225949    4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:47:12.756030 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:47:12.755992    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-znd9z" event={"ID":"2281431c-56b4-475a-a7dc-35c1bbe04e1a","Type":"ContainerStarted","Data":"d0afc0d62e8f97d2464f010d461c8190334728789c0ece70c2b3a208ee4fbbfb"}
Oct 30 22:47:12.756496 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:47:12.756068    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-znd9z" event={"ID":"2281431c-56b4-475a-a7dc-35c1bbe04e1a","Type":"ContainerStarted","Data":"28322b7823ca3b653837b0ef429897879a0aa171ea1a3f2ce72c97ed04e8df23"}
Oct 30 22:47:12.802698 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:47:12.802641    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-znd9z" podStartSLOduration=1.802623848 podStartE2EDuration="1.802623848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:47:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:47:12.799189945 +0000 UTC m=+2416.725116177" watchObservedRunningTime="2026-10-30 22:47:12.802623848 +0000 UTC m=+2416.728550054"
Oct 30 22:47:12.906857 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:47:12.906831    4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-znd9z"
Oct 30 22:47:13.909099 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:47:13.909066    4477 patch_prober.go:29] interesting pod/e2e-inf-b-867f12-openshift-default-6584f84965-znd9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body=
Oct 30 22:47:13.909500 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:47:13.909118    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-znd9z" podUID="2281431c-56b4-475a-a7dc-35c1bbe04e1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused"
Oct 30 22:47:14.907962 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:47:14.907923    4477 patch_prober.go:29] interesting pod/e2e-inf-b-867f12-openshift-default-6584f84965-znd9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body=
Oct 30 22:47:14.908170 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:47:14.907979    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-znd9z" podUID="2281431c-56b4-475a-a7dc-35c1bbe04e1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused"
Oct 30 22:47:15.907250 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:47:15.907213    4477 patch_prober.go:29] interesting pod/e2e-inf-b-867f12-openshift-default-6584f84965-znd9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body=
Oct 30 22:47:15.907703 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:47:15.907267    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-znd9z" podUID="2281431c-56b4-475a-a7dc-35c1bbe04e1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused"
Oct 30 22:47:16.907724 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:47:16.907691    4477 patch_prober.go:29] interesting pod/e2e-inf-b-867f12-openshift-default-6584f84965-znd9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body=
Oct 30 22:47:16.908127 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:47:16.907744    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-znd9z" podUID="2281431c-56b4-475a-a7dc-35c1bbe04e1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused"
Oct 30 22:47:17.907384 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:47:17.907351    4477 patch_prober.go:29] interesting pod/e2e-inf-b-867f12-openshift-default-6584f84965-znd9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body=
Oct 30 22:47:17.907565 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:47:17.907404    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-znd9z" podUID="2281431c-56b4-475a-a7dc-35c1bbe04e1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused"
Oct 30 22:47:18.926290 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:47:18.926260    4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-znd9z"
Oct 30 22:47:18.926791 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:47:18.926768    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-znd9z"
Oct 30 22:47:18.928612 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:47:18.928592    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-znd9z"
Oct 30 22:47:19.976657 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:47:19.976613    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-f0bb45/test-model-f0bb45-kserve-74fd875fb5-msp5w"]
Oct 30 22:47:26.268471 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:47:26.268429    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 22:47:35.000608 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:47:35.000564    4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-f0bb45/test-model-f0bb45-kserve-74fd875fb5-msp5w" podUID="bedea217-88bb-4871-9f6f-0eb57c9b0654" containerName="main" containerID="cri-o://4a20f7fc0a4c844ec27cd91e46fb122cfde9bcb1d0836b208d688ff38d0bf715" gracePeriod=45
Oct 30 22:47:35.283489 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:47:35.283466    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f0bb45/test-model-f0bb45-kserve-74fd875fb5-msp5w"
Oct 30 22:47:35.389206 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:47:35.389173    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bedea217-88bb-4871-9f6f-0eb57c9b0654-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bedea217-88bb-4871-9f6f-0eb57c9b0654-dshm\") pod \"bedea217-88bb-4871-9f6f-0eb57c9b0654\" (UID: \"bedea217-88bb-4871-9f6f-0eb57c9b0654\") "
Oct 30 22:47:35.389360 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:47:35.389229    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bedea217-88bb-4871-9f6f-0eb57c9b0654-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bedea217-88bb-4871-9f6f-0eb57c9b0654-tmp-dir\") pod \"bedea217-88bb-4871-9f6f-0eb57c9b0654\" (UID: \"bedea217-88bb-4871-9f6f-0eb57c9b0654\") "
Oct 30 22:47:35.389360 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:47:35.389300    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bedea217-88bb-4871-9f6f-0eb57c9b0654-tls-certs\" (UniqueName: \"kubernetes.io/secret/bedea217-88bb-4871-9f6f-0eb57c9b0654-tls-certs\") pod \"bedea217-88bb-4871-9f6f-0eb57c9b0654\" (UID: \"bedea217-88bb-4871-9f6f-0eb57c9b0654\") "
Oct 30 22:47:35.389485 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:47:35.389463    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bedea217-88bb-4871-9f6f-0eb57c9b0654-kube-api-access-kd5r9\" (UniqueName: \"kubernetes.io/projected/bedea217-88bb-4871-9f6f-0eb57c9b0654-kube-api-access-kd5r9\") pod \"bedea217-88bb-4871-9f6f-0eb57c9b0654\" (UID: \"bedea217-88bb-4871-9f6f-0eb57c9b0654\") "
Oct 30 22:47:35.389536 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:47:35.389500    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bedea217-88bb-4871-9f6f-0eb57c9b0654-tmp-dir" pod "bedea217-88bb-4871-9f6f-0eb57c9b0654" (UID: "bedea217-88bb-4871-9f6f-0eb57c9b0654"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:47:35.389536 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:47:35.389520    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bedea217-88bb-4871-9f6f-0eb57c9b0654-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bedea217-88bb-4871-9f6f-0eb57c9b0654-model-cache\") pod \"bedea217-88bb-4871-9f6f-0eb57c9b0654\" (UID: \"bedea217-88bb-4871-9f6f-0eb57c9b0654\") "
Oct 30 22:47:35.389650 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:47:35.389556    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bedea217-88bb-4871-9f6f-0eb57c9b0654-home\" (UniqueName: \"kubernetes.io/empty-dir/bedea217-88bb-4871-9f6f-0eb57c9b0654-home\") pod \"bedea217-88bb-4871-9f6f-0eb57c9b0654\" (UID: \"bedea217-88bb-4871-9f6f-0eb57c9b0654\") "
Oct 30 22:47:35.389845 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:47:35.389785    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bedea217-88bb-4871-9f6f-0eb57c9b0654-model-cache" pod "bedea217-88bb-4871-9f6f-0eb57c9b0654" (UID: "bedea217-88bb-4871-9f6f-0eb57c9b0654"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:47:35.389978 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:47:35.389888    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bedea217-88bb-4871-9f6f-0eb57c9b0654-home" pod "bedea217-88bb-4871-9f6f-0eb57c9b0654" (UID: "bedea217-88bb-4871-9f6f-0eb57c9b0654"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:47:35.390262 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:47:35.390240    4477 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bedea217-88bb-4871-9f6f-0eb57c9b0654-tmp-dir\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:47:35.390321 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:47:35.390270    4477 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bedea217-88bb-4871-9f6f-0eb57c9b0654-model-cache\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:47:35.390321 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:47:35.390285    4477 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bedea217-88bb-4871-9f6f-0eb57c9b0654-home\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:47:35.391695 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:47:35.391666    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bedea217-88bb-4871-9f6f-0eb57c9b0654-tls-certs" pod "bedea217-88bb-4871-9f6f-0eb57c9b0654" (UID: "bedea217-88bb-4871-9f6f-0eb57c9b0654"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:47:35.391831 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:47:35.391807    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bedea217-88bb-4871-9f6f-0eb57c9b0654-dshm" pod "bedea217-88bb-4871-9f6f-0eb57c9b0654" (UID: "bedea217-88bb-4871-9f6f-0eb57c9b0654"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:47:35.391933 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:47:35.391859    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bedea217-88bb-4871-9f6f-0eb57c9b0654-kube-api-access-kd5r9" pod "bedea217-88bb-4871-9f6f-0eb57c9b0654" (UID: "bedea217-88bb-4871-9f6f-0eb57c9b0654"). InnerVolumeSpecName "kube-api-access-kd5r9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:47:35.491654 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:47:35.491627    4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kd5r9\" (UniqueName: \"kubernetes.io/projected/bedea217-88bb-4871-9f6f-0eb57c9b0654-kube-api-access-kd5r9\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:47:35.491654 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:47:35.491653    4477 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bedea217-88bb-4871-9f6f-0eb57c9b0654-dshm\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:47:35.491810 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:47:35.491666    4477 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bedea217-88bb-4871-9f6f-0eb57c9b0654-tls-certs\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:47:35.924448 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:47:35.924414    4477 generic.go:356] "Generic (PLEG): container finished" podID="bedea217-88bb-4871-9f6f-0eb57c9b0654" containerID="4a20f7fc0a4c844ec27cd91e46fb122cfde9bcb1d0836b208d688ff38d0bf715" exitCode=0
Oct 30 22:47:35.924597 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:47:35.924451    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f0bb45/test-model-f0bb45-kserve-74fd875fb5-msp5w" event={"ID":"bedea217-88bb-4871-9f6f-0eb57c9b0654","Type":"ContainerDied","Data":"4a20f7fc0a4c844ec27cd91e46fb122cfde9bcb1d0836b208d688ff38d0bf715"}
Oct 30 22:47:35.924597 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:47:35.924498    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f0bb45/test-model-f0bb45-kserve-74fd875fb5-msp5w" event={"ID":"bedea217-88bb-4871-9f6f-0eb57c9b0654","Type":"ContainerDied","Data":"c507462ed9a8ee6b2c15ab4d0412b2f8589b1649f060fb6df5d4860cae6fd7d0"}
Oct 30 22:47:35.924597 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:47:35.924515    4477 scope.go:122] "RemoveContainer" containerID="4a20f7fc0a4c844ec27cd91e46fb122cfde9bcb1d0836b208d688ff38d0bf715"
Oct 30 22:47:35.924597 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:47:35.924460    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f0bb45/test-model-f0bb45-kserve-74fd875fb5-msp5w"
Oct 30 22:47:35.935565 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:47:35.935549    4477 scope.go:122] "RemoveContainer" containerID="4a20f7fc0a4c844ec27cd91e46fb122cfde9bcb1d0836b208d688ff38d0bf715"
Oct 30 22:47:35.935823 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:47:35.935805    4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a20f7fc0a4c844ec27cd91e46fb122cfde9bcb1d0836b208d688ff38d0bf715\": container with ID starting with 4a20f7fc0a4c844ec27cd91e46fb122cfde9bcb1d0836b208d688ff38d0bf715 not found: ID does not exist" containerID="4a20f7fc0a4c844ec27cd91e46fb122cfde9bcb1d0836b208d688ff38d0bf715"
Oct 30 22:47:35.935868 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:47:35.935831    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a20f7fc0a4c844ec27cd91e46fb122cfde9bcb1d0836b208d688ff38d0bf715"} err="failed to get container status \"4a20f7fc0a4c844ec27cd91e46fb122cfde9bcb1d0836b208d688ff38d0bf715\": rpc error: code = NotFound desc = could not find container \"4a20f7fc0a4c844ec27cd91e46fb122cfde9bcb1d0836b208d688ff38d0bf715\": container with ID starting with 4a20f7fc0a4c844ec27cd91e46fb122cfde9bcb1d0836b208d688ff38d0bf715 not found: ID does not exist"
Oct 30 22:47:35.976169 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:47:35.976132    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-f0bb45/test-model-f0bb45-kserve-74fd875fb5-msp5w"]
Oct 30 22:47:35.987306 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:47:35.987261    4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-f0bb45/test-model-f0bb45-kserve-74fd875fb5-msp5w"]
Oct 30 22:47:36.196643 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:47:36.196567    4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bedea217-88bb-4871-9f6f-0eb57c9b0654" path="/var/lib/kubelet/pods/bedea217-88bb-4871-9f6f-0eb57c9b0654/volumes"
Oct 30 22:47:56.274872 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:47:56.274831    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 22:48:03.540182 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:03.540147    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-40229e-f6c86585d-gdhkl"]
Oct 30 22:48:03.540657 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:03.540332    4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-40229e-f6c86585d-gdhkl" podUID="8c927c80-1da0-4b5b-8655-c67e8a35f000" containerName="maas-api" containerID="cri-o://664b4ceebfaa55cb32f797d5d2b9d53b7fd0eab04588173a390c2d58e450fc47" gracePeriod=30
Oct 30 22:48:03.984891 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:03.979870    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-54c7f0-84c875b494-fqqw8"]
Oct 30 22:48:03.996062 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:03.993170    4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-54c7f0-84c875b494-fqqw8" podUID="b47ab287-9d84-468d-8afb-28b3f6b862e2" containerName="maas-api" containerID="cri-o://d39669020ab4c553387b31855567432e44bfda069c766dfcb3e003361bfc2c05" gracePeriod=30
Oct 30 22:48:04.033824 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:04.027697    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-40229e-f6c86585d-gdhkl"
Oct 30 22:48:04.096958 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:04.095646    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c927c80-1da0-4b5b-8655-c67e8a35f000-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c927c80-1da0-4b5b-8655-c67e8a35f000-openshift-service-ca\") pod \"8c927c80-1da0-4b5b-8655-c67e8a35f000\" (UID: \"8c927c80-1da0-4b5b-8655-c67e8a35f000\") "
Oct 30 22:48:04.096958 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:04.095914    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c927c80-1da0-4b5b-8655-c67e8a35f000-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c927c80-1da0-4b5b-8655-c67e8a35f000-metrics-certs\") pod \"8c927c80-1da0-4b5b-8655-c67e8a35f000\" (UID: \"8c927c80-1da0-4b5b-8655-c67e8a35f000\") "
Oct 30 22:48:04.096958 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:04.095971    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c927c80-1da0-4b5b-8655-c67e8a35f000-kube-api-access-mbkrd\" (UniqueName: \"kubernetes.io/projected/8c927c80-1da0-4b5b-8655-c67e8a35f000-kube-api-access-mbkrd\") pod \"8c927c80-1da0-4b5b-8655-c67e8a35f000\" (UID: \"8c927c80-1da0-4b5b-8655-c67e8a35f000\") "
Oct 30 22:48:04.096958 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:04.096020    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c927c80-1da0-4b5b-8655-c67e8a35f000-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c927c80-1da0-4b5b-8655-c67e8a35f000-maas-api-tls\") pod \"8c927c80-1da0-4b5b-8655-c67e8a35f000\" (UID: \"8c927c80-1da0-4b5b-8655-c67e8a35f000\") "
Oct 30 22:48:04.098334 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:04.098308    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c927c80-1da0-4b5b-8655-c67e8a35f000-openshift-service-ca" pod "8c927c80-1da0-4b5b-8655-c67e8a35f000" (UID: "8c927c80-1da0-4b5b-8655-c67e8a35f000"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:48:04.103161 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:04.103133    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c927c80-1da0-4b5b-8655-c67e8a35f000-kube-api-access-mbkrd" pod "8c927c80-1da0-4b5b-8655-c67e8a35f000" (UID: "8c927c80-1da0-4b5b-8655-c67e8a35f000"). InnerVolumeSpecName "kube-api-access-mbkrd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:48:04.114338 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:04.114306    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c927c80-1da0-4b5b-8655-c67e8a35f000-metrics-certs" pod "8c927c80-1da0-4b5b-8655-c67e8a35f000" (UID: "8c927c80-1da0-4b5b-8655-c67e8a35f000"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:48:04.124366 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:04.123771    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c927c80-1da0-4b5b-8655-c67e8a35f000-maas-api-tls" pod "8c927c80-1da0-4b5b-8655-c67e8a35f000" (UID: "8c927c80-1da0-4b5b-8655-c67e8a35f000"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:48:04.142534 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:04.142448    4477 generic.go:356] "Generic (PLEG): container finished" podID="8c927c80-1da0-4b5b-8655-c67e8a35f000" containerID="664b4ceebfaa55cb32f797d5d2b9d53b7fd0eab04588173a390c2d58e450fc47" exitCode=0
Oct 30 22:48:04.142682 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:04.142548    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-40229e-f6c86585d-gdhkl" event={"ID":"8c927c80-1da0-4b5b-8655-c67e8a35f000","Type":"ContainerDied","Data":"664b4ceebfaa55cb32f797d5d2b9d53b7fd0eab04588173a390c2d58e450fc47"}
Oct 30 22:48:04.142682 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:04.142603    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-40229e-f6c86585d-gdhkl" event={"ID":"8c927c80-1da0-4b5b-8655-c67e8a35f000","Type":"ContainerDied","Data":"741f2711cf8d427ef6e6d33d9d379de35473b55ef6649938809619ba9d3ac052"}
Oct 30 22:48:04.142682 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:04.142625    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-40229e-f6c86585d-gdhkl"
Oct 30 22:48:04.142890 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:04.142635    4477 scope.go:122] "RemoveContainer" containerID="664b4ceebfaa55cb32f797d5d2b9d53b7fd0eab04588173a390c2d58e450fc47"
Oct 30 22:48:04.205933 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:04.205243    4477 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c927c80-1da0-4b5b-8655-c67e8a35f000-openshift-service-ca\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:48:04.205933 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:04.205276    4477 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c927c80-1da0-4b5b-8655-c67e8a35f000-metrics-certs\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:48:04.205933 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:04.205292    4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbkrd\" (UniqueName: \"kubernetes.io/projected/8c927c80-1da0-4b5b-8655-c67e8a35f000-kube-api-access-mbkrd\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:48:04.205933 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:04.205304    4477 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c927c80-1da0-4b5b-8655-c67e8a35f000-maas-api-tls\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:48:04.220764 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:04.220735    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-40229e-f6c86585d-gdhkl"]
Oct 30 22:48:04.225907 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:04.225146    4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-40229e-f6c86585d-gdhkl"]
Oct 30 22:48:04.228423 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:04.228403    4477 scope.go:122] "RemoveContainer" containerID="664b4ceebfaa55cb32f797d5d2b9d53b7fd0eab04588173a390c2d58e450fc47"
Oct 30 22:48:04.228905 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:48:04.228878    4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"664b4ceebfaa55cb32f797d5d2b9d53b7fd0eab04588173a390c2d58e450fc47\": container with ID starting with 664b4ceebfaa55cb32f797d5d2b9d53b7fd0eab04588173a390c2d58e450fc47 not found: ID does not exist" containerID="664b4ceebfaa55cb32f797d5d2b9d53b7fd0eab04588173a390c2d58e450fc47"
Oct 30 22:48:04.229009 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:04.228906    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"664b4ceebfaa55cb32f797d5d2b9d53b7fd0eab04588173a390c2d58e450fc47"} err="failed to get container status \"664b4ceebfaa55cb32f797d5d2b9d53b7fd0eab04588173a390c2d58e450fc47\": rpc error: code = NotFound desc = could not find container \"664b4ceebfaa55cb32f797d5d2b9d53b7fd0eab04588173a390c2d58e450fc47\": container with ID starting with 664b4ceebfaa55cb32f797d5d2b9d53b7fd0eab04588173a390c2d58e450fc47 not found: ID does not exist"
Oct 30 22:48:04.371317 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:04.371282    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-54c7f0-84c875b494-fqqw8"
Oct 30 22:48:04.411560 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:04.411477    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b47ab287-9d84-468d-8afb-28b3f6b862e2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b47ab287-9d84-468d-8afb-28b3f6b862e2-metrics-certs\") pod \"b47ab287-9d84-468d-8afb-28b3f6b862e2\" (UID: \"b47ab287-9d84-468d-8afb-28b3f6b862e2\") "
Oct 30 22:48:04.432671 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:04.422494    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b47ab287-9d84-468d-8afb-28b3f6b862e2-metrics-certs" pod "b47ab287-9d84-468d-8afb-28b3f6b862e2" (UID: "b47ab287-9d84-468d-8afb-28b3f6b862e2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:48:04.433153 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:04.433128    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b47ab287-9d84-468d-8afb-28b3f6b862e2-kube-api-access-tnp2t\" (UniqueName: \"kubernetes.io/projected/b47ab287-9d84-468d-8afb-28b3f6b862e2-kube-api-access-tnp2t\") pod \"b47ab287-9d84-468d-8afb-28b3f6b862e2\" (UID: \"b47ab287-9d84-468d-8afb-28b3f6b862e2\") "
Oct 30 22:48:04.433415 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:04.433399    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b47ab287-9d84-468d-8afb-28b3f6b862e2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b47ab287-9d84-468d-8afb-28b3f6b862e2-openshift-service-ca\") pod \"b47ab287-9d84-468d-8afb-28b3f6b862e2\" (UID: \"b47ab287-9d84-468d-8afb-28b3f6b862e2\") "
Oct 30 22:48:04.433620 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:04.433605    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b47ab287-9d84-468d-8afb-28b3f6b862e2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b47ab287-9d84-468d-8afb-28b3f6b862e2-maas-api-tls\") pod \"b47ab287-9d84-468d-8afb-28b3f6b862e2\" (UID: \"b47ab287-9d84-468d-8afb-28b3f6b862e2\") "
Oct 30 22:48:04.434447 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:04.434415    4477 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b47ab287-9d84-468d-8afb-28b3f6b862e2-metrics-certs\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:48:04.436014 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:04.435978    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b47ab287-9d84-468d-8afb-28b3f6b862e2-openshift-service-ca" pod "b47ab287-9d84-468d-8afb-28b3f6b862e2" (UID: "b47ab287-9d84-468d-8afb-28b3f6b862e2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:48:04.437365 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:04.437340    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b47ab287-9d84-468d-8afb-28b3f6b862e2-kube-api-access-tnp2t" pod "b47ab287-9d84-468d-8afb-28b3f6b862e2" (UID: "b47ab287-9d84-468d-8afb-28b3f6b862e2"). InnerVolumeSpecName "kube-api-access-tnp2t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:48:04.448065 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:04.447521    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b47ab287-9d84-468d-8afb-28b3f6b862e2-maas-api-tls" pod "b47ab287-9d84-468d-8afb-28b3f6b862e2" (UID: "b47ab287-9d84-468d-8afb-28b3f6b862e2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:48:04.536692 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:04.536651    4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnp2t\" (UniqueName: \"kubernetes.io/projected/b47ab287-9d84-468d-8afb-28b3f6b862e2-kube-api-access-tnp2t\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:48:04.536692 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:04.536692    4477 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b47ab287-9d84-468d-8afb-28b3f6b862e2-openshift-service-ca\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:48:04.536893 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:04.536703    4477 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b47ab287-9d84-468d-8afb-28b3f6b862e2-maas-api-tls\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:48:05.151176 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:05.151144    4477 generic.go:356] "Generic (PLEG): container finished" podID="b47ab287-9d84-468d-8afb-28b3f6b862e2" containerID="d39669020ab4c553387b31855567432e44bfda069c766dfcb3e003361bfc2c05" exitCode=0
Oct 30 22:48:05.151631 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:05.151200    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-54c7f0-84c875b494-fqqw8"
Oct 30 22:48:05.151631 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:05.151225    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-54c7f0-84c875b494-fqqw8" event={"ID":"b47ab287-9d84-468d-8afb-28b3f6b862e2","Type":"ContainerDied","Data":"d39669020ab4c553387b31855567432e44bfda069c766dfcb3e003361bfc2c05"}
Oct 30 22:48:05.151631 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:05.151276    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-54c7f0-84c875b494-fqqw8" event={"ID":"b47ab287-9d84-468d-8afb-28b3f6b862e2","Type":"ContainerDied","Data":"6ec1716f47ec75e0c60dcadb60fc3ae548996d5bb421663383175b67c35d9700"}
Oct 30 22:48:05.151631 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:05.151300    4477 scope.go:122] "RemoveContainer" containerID="d39669020ab4c553387b31855567432e44bfda069c766dfcb3e003361bfc2c05"
Oct 30 22:48:05.162506 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:05.162488    4477 scope.go:122] "RemoveContainer" containerID="d39669020ab4c553387b31855567432e44bfda069c766dfcb3e003361bfc2c05"
Oct 30 22:48:05.162818 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:48:05.162792    4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d39669020ab4c553387b31855567432e44bfda069c766dfcb3e003361bfc2c05\": container with ID starting with d39669020ab4c553387b31855567432e44bfda069c766dfcb3e003361bfc2c05 not found: ID does not exist" containerID="d39669020ab4c553387b31855567432e44bfda069c766dfcb3e003361bfc2c05"
Oct 30 22:48:05.162960 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:05.162826    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d39669020ab4c553387b31855567432e44bfda069c766dfcb3e003361bfc2c05"} err="failed to get container status \"d39669020ab4c553387b31855567432e44bfda069c766dfcb3e003361bfc2c05\": rpc error: code = NotFound desc = could not find container \"d39669020ab4c553387b31855567432e44bfda069c766dfcb3e003361bfc2c05\": container with ID starting with d39669020ab4c553387b31855567432e44bfda069c766dfcb3e003361bfc2c05 not found: ID does not exist"
Oct 30 22:48:05.201908 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:05.201879    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-54c7f0-84c875b494-fqqw8"]
Oct 30 22:48:05.213102 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:05.213081    4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-54c7f0-84c875b494-fqqw8"]
Oct 30 22:48:06.194168 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:06.194121    4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c927c80-1da0-4b5b-8655-c67e8a35f000" path="/var/lib/kubelet/pods/8c927c80-1da0-4b5b-8655-c67e8a35f000/volumes"
Oct 30 22:48:06.194615 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:06.194600    4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b47ab287-9d84-468d-8afb-28b3f6b862e2" path="/var/lib/kubelet/pods/b47ab287-9d84-468d-8afb-28b3f6b862e2/volumes"
Oct 30 22:48:07.019919 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:07.019869    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-40229e-f6c86585d-gdhkl" podUID="8c927c80-1da0-4b5b-8655-c67e8a35f000" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.117:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:48:19.237456 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:19.237419    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-3acfdb/test-model-3acfdb-kserve-79856cb545-b29mk"]
Oct 30 22:48:26.268952 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:48:26.268914    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 22:48:31.985403 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:31.985370    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f0bb45-fbdcb4dd6-pvjvt"]
Oct 30 22:48:31.986819 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:31.985588    4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f0bb45-fbdcb4dd6-pvjvt" podUID="616d47a3-a660-48d3-9714-9e0f45a5660a" containerName="maas-api" containerID="cri-o://3b091c37446e46d81a03d352931431ae28cf96b5997c11435a783b49bc2ddee4" gracePeriod=30
Oct 30 22:48:32.339073 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:32.337303    4477 generic.go:356] "Generic (PLEG): container finished" podID="616d47a3-a660-48d3-9714-9e0f45a5660a" containerID="3b091c37446e46d81a03d352931431ae28cf96b5997c11435a783b49bc2ddee4" exitCode=0
Oct 30 22:48:32.339073 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:32.337375    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f0bb45-fbdcb4dd6-pvjvt" event={"ID":"616d47a3-a660-48d3-9714-9e0f45a5660a","Type":"ContainerDied","Data":"3b091c37446e46d81a03d352931431ae28cf96b5997c11435a783b49bc2ddee4"}
Oct 30 22:48:32.339073 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:32.337423    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f0bb45-fbdcb4dd6-pvjvt" event={"ID":"616d47a3-a660-48d3-9714-9e0f45a5660a","Type":"ContainerDied","Data":"cf628afec8a1c2982ed7def8a7d8f76522cd41efb3e97d33e7203a89cc59f2a7"}
Oct 30 22:48:32.339073 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:32.337444    4477 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cf628afec8a1c2982ed7def8a7d8f76522cd41efb3e97d33e7203a89cc59f2a7"
Oct 30 22:48:32.349574 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:32.349546    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f0bb45-fbdcb4dd6-pvjvt"
Oct 30 22:48:32.436679 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:32.434992    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/616d47a3-a660-48d3-9714-9e0f45a5660a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/616d47a3-a660-48d3-9714-9e0f45a5660a-maas-api-tls\") pod \"616d47a3-a660-48d3-9714-9e0f45a5660a\" (UID: \"616d47a3-a660-48d3-9714-9e0f45a5660a\") "
Oct 30 22:48:32.436679 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:32.435346    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/616d47a3-a660-48d3-9714-9e0f45a5660a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/616d47a3-a660-48d3-9714-9e0f45a5660a-metrics-certs\") pod \"616d47a3-a660-48d3-9714-9e0f45a5660a\" (UID: \"616d47a3-a660-48d3-9714-9e0f45a5660a\") "
Oct 30 22:48:32.436679 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:32.435400    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/616d47a3-a660-48d3-9714-9e0f45a5660a-kube-api-access-szwtw\" (UniqueName: \"kubernetes.io/projected/616d47a3-a660-48d3-9714-9e0f45a5660a-kube-api-access-szwtw\") pod \"616d47a3-a660-48d3-9714-9e0f45a5660a\" (UID: \"616d47a3-a660-48d3-9714-9e0f45a5660a\") "
Oct 30 22:48:32.436679 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:32.435448    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/616d47a3-a660-48d3-9714-9e0f45a5660a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/616d47a3-a660-48d3-9714-9e0f45a5660a-openshift-service-ca\") pod \"616d47a3-a660-48d3-9714-9e0f45a5660a\" (UID: \"616d47a3-a660-48d3-9714-9e0f45a5660a\") "
Oct 30 22:48:32.440059 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:32.437972    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/616d47a3-a660-48d3-9714-9e0f45a5660a-openshift-service-ca" pod "616d47a3-a660-48d3-9714-9e0f45a5660a" (UID: "616d47a3-a660-48d3-9714-9e0f45a5660a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:48:32.467874 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:32.466495    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/616d47a3-a660-48d3-9714-9e0f45a5660a-kube-api-access-szwtw" pod "616d47a3-a660-48d3-9714-9e0f45a5660a" (UID: "616d47a3-a660-48d3-9714-9e0f45a5660a"). InnerVolumeSpecName "kube-api-access-szwtw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:48:32.498171 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:32.496609    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/616d47a3-a660-48d3-9714-9e0f45a5660a-maas-api-tls" pod "616d47a3-a660-48d3-9714-9e0f45a5660a" (UID: "616d47a3-a660-48d3-9714-9e0f45a5660a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:48:32.498632 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:32.498598    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/616d47a3-a660-48d3-9714-9e0f45a5660a-metrics-certs" pod "616d47a3-a660-48d3-9714-9e0f45a5660a" (UID: "616d47a3-a660-48d3-9714-9e0f45a5660a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:48:32.538385 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:32.538272    4477 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/616d47a3-a660-48d3-9714-9e0f45a5660a-metrics-certs\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:48:32.538385 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:32.538306    4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szwtw\" (UniqueName: \"kubernetes.io/projected/616d47a3-a660-48d3-9714-9e0f45a5660a-kube-api-access-szwtw\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:48:32.538385 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:32.538323    4477 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/616d47a3-a660-48d3-9714-9e0f45a5660a-openshift-service-ca\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:48:32.538385 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:32.538337    4477 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/616d47a3-a660-48d3-9714-9e0f45a5660a-maas-api-tls\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:48:33.343887 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:33.343815    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-f0bb45-fbdcb4dd6-pvjvt"
Oct 30 22:48:33.395570 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:33.395538    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f0bb45-fbdcb4dd6-pvjvt"]
Oct 30 22:48:33.406638 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:33.406612    4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-f0bb45-fbdcb4dd6-pvjvt"]
Oct 30 22:48:34.193666 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:34.193627    4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="616d47a3-a660-48d3-9714-9e0f45a5660a" path="/var/lib/kubelet/pods/616d47a3-a660-48d3-9714-9e0f45a5660a/volumes"
Oct 30 22:48:34.272626 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:34.272587    4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-3acfdb/test-model-3acfdb-kserve-79856cb545-b29mk" podUID="13dfee80-6a6d-4b97-be9d-728a85a59e5b" containerName="main" containerID="cri-o://0ffeeda2858fda9dd9a7a262cede77cd24788736d17a4dccb164492c3773d640" gracePeriod=45
Oct 30 22:48:34.351002 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:34.350975    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-3acfdb/test-model-3acfdb-kserve-79856cb545-b29mk" podUID="13dfee80-6a6d-4b97-be9d-728a85a59e5b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.129:8000/ready\": dial tcp 10.134.0.129:8000: connect: connection refused"
Oct 30 22:48:34.540481 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:34.540458    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-3acfdb/test-model-3acfdb-kserve-79856cb545-b29mk"
Oct 30 22:48:34.664506 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:34.664474    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13dfee80-6a6d-4b97-be9d-728a85a59e5b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13dfee80-6a6d-4b97-be9d-728a85a59e5b-model-cache\") pod \"13dfee80-6a6d-4b97-be9d-728a85a59e5b\" (UID: \"13dfee80-6a6d-4b97-be9d-728a85a59e5b\") "
Oct 30 22:48:34.664687 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:34.664558    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13dfee80-6a6d-4b97-be9d-728a85a59e5b-kube-api-access-l2mgv\" (UniqueName: \"kubernetes.io/projected/13dfee80-6a6d-4b97-be9d-728a85a59e5b-kube-api-access-l2mgv\") pod \"13dfee80-6a6d-4b97-be9d-728a85a59e5b\" (UID: \"13dfee80-6a6d-4b97-be9d-728a85a59e5b\") "
Oct 30 22:48:34.664754 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:34.664740    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13dfee80-6a6d-4b97-be9d-728a85a59e5b-tls-certs\" (UniqueName: \"kubernetes.io/secret/13dfee80-6a6d-4b97-be9d-728a85a59e5b-tls-certs\") pod \"13dfee80-6a6d-4b97-be9d-728a85a59e5b\" (UID: \"13dfee80-6a6d-4b97-be9d-728a85a59e5b\") "
Oct 30 22:48:34.664824 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:34.664744    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13dfee80-6a6d-4b97-be9d-728a85a59e5b-model-cache" pod "13dfee80-6a6d-4b97-be9d-728a85a59e5b" (UID: "13dfee80-6a6d-4b97-be9d-728a85a59e5b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:48:34.664824 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:34.664782    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13dfee80-6a6d-4b97-be9d-728a85a59e5b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/13dfee80-6a6d-4b97-be9d-728a85a59e5b-dshm\") pod \"13dfee80-6a6d-4b97-be9d-728a85a59e5b\" (UID: \"13dfee80-6a6d-4b97-be9d-728a85a59e5b\") "
Oct 30 22:48:34.664930 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:34.664885    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13dfee80-6a6d-4b97-be9d-728a85a59e5b-home\" (UniqueName: \"kubernetes.io/empty-dir/13dfee80-6a6d-4b97-be9d-728a85a59e5b-home\") pod \"13dfee80-6a6d-4b97-be9d-728a85a59e5b\" (UID: \"13dfee80-6a6d-4b97-be9d-728a85a59e5b\") "
Oct 30 22:48:34.664930 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:34.664917    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13dfee80-6a6d-4b97-be9d-728a85a59e5b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13dfee80-6a6d-4b97-be9d-728a85a59e5b-tmp-dir\") pod \"13dfee80-6a6d-4b97-be9d-728a85a59e5b\" (UID: \"13dfee80-6a6d-4b97-be9d-728a85a59e5b\") "
Oct 30 22:48:34.665244 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:34.665219    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13dfee80-6a6d-4b97-be9d-728a85a59e5b-home" pod "13dfee80-6a6d-4b97-be9d-728a85a59e5b" (UID: "13dfee80-6a6d-4b97-be9d-728a85a59e5b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:48:34.665374 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:34.665261    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13dfee80-6a6d-4b97-be9d-728a85a59e5b-tmp-dir" pod "13dfee80-6a6d-4b97-be9d-728a85a59e5b" (UID: "13dfee80-6a6d-4b97-be9d-728a85a59e5b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:48:34.665510 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:34.665493    4477 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/13dfee80-6a6d-4b97-be9d-728a85a59e5b-home\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:48:34.665582 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:34.665516    4477 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/13dfee80-6a6d-4b97-be9d-728a85a59e5b-tmp-dir\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:48:34.665582 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:34.665527    4477 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/13dfee80-6a6d-4b97-be9d-728a85a59e5b-model-cache\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:48:34.667159 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:34.667132    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13dfee80-6a6d-4b97-be9d-728a85a59e5b-tls-certs" pod "13dfee80-6a6d-4b97-be9d-728a85a59e5b" (UID: "13dfee80-6a6d-4b97-be9d-728a85a59e5b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:48:34.667265 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:34.667187    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13dfee80-6a6d-4b97-be9d-728a85a59e5b-kube-api-access-l2mgv" pod "13dfee80-6a6d-4b97-be9d-728a85a59e5b" (UID: "13dfee80-6a6d-4b97-be9d-728a85a59e5b"). InnerVolumeSpecName "kube-api-access-l2mgv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:48:34.667400 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:34.667378    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13dfee80-6a6d-4b97-be9d-728a85a59e5b-dshm" pod "13dfee80-6a6d-4b97-be9d-728a85a59e5b" (UID: "13dfee80-6a6d-4b97-be9d-728a85a59e5b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:48:34.767001 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:34.766919    4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2mgv\" (UniqueName: \"kubernetes.io/projected/13dfee80-6a6d-4b97-be9d-728a85a59e5b-kube-api-access-l2mgv\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:48:34.767001 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:34.766950    4477 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/13dfee80-6a6d-4b97-be9d-728a85a59e5b-tls-certs\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:48:34.767001 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:34.766960    4477 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/13dfee80-6a6d-4b97-be9d-728a85a59e5b-dshm\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:48:35.366062 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:35.361710    4477 generic.go:356] "Generic (PLEG): container finished" podID="13dfee80-6a6d-4b97-be9d-728a85a59e5b" containerID="0ffeeda2858fda9dd9a7a262cede77cd24788736d17a4dccb164492c3773d640" exitCode=0
Oct 30 22:48:35.366062 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:35.361851    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-3acfdb/test-model-3acfdb-kserve-79856cb545-b29mk"
Oct 30 22:48:35.366062 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:35.361763    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-3acfdb/test-model-3acfdb-kserve-79856cb545-b29mk" event={"ID":"13dfee80-6a6d-4b97-be9d-728a85a59e5b","Type":"ContainerDied","Data":"0ffeeda2858fda9dd9a7a262cede77cd24788736d17a4dccb164492c3773d640"}
Oct 30 22:48:35.366062 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:35.361996    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-3acfdb/test-model-3acfdb-kserve-79856cb545-b29mk" event={"ID":"13dfee80-6a6d-4b97-be9d-728a85a59e5b","Type":"ContainerDied","Data":"7184f8efcaf96b290014d3cd8ddaf61a48fe59a4e7bab29d45e2953332cee526"}
Oct 30 22:48:35.366062 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:35.362021    4477 scope.go:122] "RemoveContainer" containerID="0ffeeda2858fda9dd9a7a262cede77cd24788736d17a4dccb164492c3773d640"
Oct 30 22:48:35.381964 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:35.379949    4477 scope.go:122] "RemoveContainer" containerID="0ffeeda2858fda9dd9a7a262cede77cd24788736d17a4dccb164492c3773d640"
Oct 30 22:48:35.381964 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:48:35.381420    4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ffeeda2858fda9dd9a7a262cede77cd24788736d17a4dccb164492c3773d640\": container with ID starting with 0ffeeda2858fda9dd9a7a262cede77cd24788736d17a4dccb164492c3773d640 not found: ID does not exist" containerID="0ffeeda2858fda9dd9a7a262cede77cd24788736d17a4dccb164492c3773d640"
Oct 30 22:48:35.381964 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:35.381447    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ffeeda2858fda9dd9a7a262cede77cd24788736d17a4dccb164492c3773d640"} err="failed to get container status \"0ffeeda2858fda9dd9a7a262cede77cd24788736d17a4dccb164492c3773d640\": rpc error: code = NotFound desc = could not find container \"0ffeeda2858fda9dd9a7a262cede77cd24788736d17a4dccb164492c3773d640\": container with ID starting with 0ffeeda2858fda9dd9a7a262cede77cd24788736d17a4dccb164492c3773d640 not found: ID does not exist"
Oct 30 22:48:35.509063 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:35.508883    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-3acfdb/test-model-3acfdb-kserve-79856cb545-b29mk"]
Oct 30 22:48:35.523167 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:35.523127    4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-3acfdb/test-model-3acfdb-kserve-79856cb545-b29mk"]
Oct 30 22:48:36.194324 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:36.194292    4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13dfee80-6a6d-4b97-be9d-728a85a59e5b" path="/var/lib/kubelet/pods/13dfee80-6a6d-4b97-be9d-728a85a59e5b/volumes"
Oct 30 22:48:42.181838 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:42.181802    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hq9vz"]
Oct 30 22:48:42.182337 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:42.182023    4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hq9vz" podUID="76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1" containerName="istio-proxy" containerID="cri-o://db69e1b6f0249022202674e28dc9b17682d88f0154d4960524be464b4e7046e4" gracePeriod=30
Oct 30 22:48:42.384091 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:42.384058    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-lg99w"]
Oct 30 22:48:42.384310 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:42.384268    4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-lg99w" podUID="8e7915fd-e76a-407d-825e-ae095f5d3c56" containerName="istio-proxy" containerID="cri-o://de5712116b50a298f8a609b6ca5f0b7b67659e2cfd6c70cb4060fe984f6318bc" gracePeriod=30
Oct 30 22:48:46.153493 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:46.153453    4477 patch_prober.go:29] interesting pod/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hq9vz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:48:46.153871 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:46.153507    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hq9vz" podUID="76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:48:46.489243 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:46.489161    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6bz92"]
Oct 30 22:48:47.126989 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:47.126951    4477 patch_prober.go:29] interesting pod/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-lg99w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:48:47.127201 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:47.127008    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-lg99w" podUID="8e7915fd-e76a-407d-825e-ae095f5d3c56" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:48:47.466611 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:47.466577    4477 generic.go:356] "Generic (PLEG): container finished" podID="76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1" containerID="db69e1b6f0249022202674e28dc9b17682d88f0154d4960524be464b4e7046e4" exitCode=0
Oct 30 22:48:47.467161 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:47.467136    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hq9vz" event={"ID":"76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1","Type":"ContainerDied","Data":"db69e1b6f0249022202674e28dc9b17682d88f0154d4960524be464b4e7046e4"}
Oct 30 22:48:47.667890 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:47.667553    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hq9vz"
Oct 30 22:48:47.782024 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:47.781995    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-lg99w"
Oct 30 22:48:47.837140 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:47.837111    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1-kube-api-access-d9whg\" (UniqueName: \"kubernetes.io/projected/76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1-kube-api-access-d9whg\") pod \"76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1\" (UID: \"76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1\") "
Oct 30 22:48:47.837311 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:47.837164    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1-istio-envoy\") pod \"76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1\" (UID: \"76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1\") "
Oct 30 22:48:47.837311 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:47.837257    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1-istio-data\") pod \"76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1\" (UID: \"76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1\") "
Oct 30 22:48:47.837311 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:47.837297    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1-istio-podinfo\") pod \"76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1\" (UID: \"76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1\") "
Oct 30 22:48:47.837496 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:47.837384    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1-workload-certs\") pod \"76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1\" (UID: \"76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1\") "
Oct 30 22:48:47.837496 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:47.837459    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1-istio-token\" (UniqueName: \"kubernetes.io/projected/76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1-istio-token\") pod \"76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1\" (UID: \"76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1\") "
Oct 30 22:48:47.837601 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:47.837511    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1-credential-socket\") pod \"76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1\" (UID: \"76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1\") "
Oct 30 22:48:47.837601 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:47.837581    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1-workload-socket\") pod \"76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1\" (UID: \"76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1\") "
Oct 30 22:48:47.837601 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:47.837585    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1-istio-data" pod "76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1" (UID: "76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:48:47.837757 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:47.837651    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1-istiod-ca-cert\") pod \"76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1\" (UID: \"76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1\") "
Oct 30 22:48:47.837863 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:47.837836    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1-workload-certs" pod "76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1" (UID: "76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:48:47.838145 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:47.838120    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1-credential-socket" pod "76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1" (UID: "76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:48:47.838497 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:47.838476    4477 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1-istio-data\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:48:47.838639 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:47.838624    4477 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1-workload-certs\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:48:47.838778 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:47.838763    4477 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1-credential-socket\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:48:47.838924 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:47.838529    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1-workload-socket" pod "76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1" (UID: "76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:48:47.839078 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:47.838581    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1-istiod-ca-cert" pod "76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1" (UID: "76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:48:47.840009 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:47.839981    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1-kube-api-access-d9whg" pod "76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1" (UID: "76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1"). InnerVolumeSpecName "kube-api-access-d9whg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:48:47.840290 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:47.840258    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1-istio-envoy" pod "76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1" (UID: "76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:48:47.840451 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:47.840432    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1-istio-podinfo" pod "76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1" (UID: "76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:48:47.847502 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:47.847475    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1-istio-token" pod "76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1" (UID: "76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:48:47.940003 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:47.939974    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e7915fd-e76a-407d-825e-ae095f5d3c56-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e7915fd-e76a-407d-825e-ae095f5d3c56-workload-certs\") pod \"8e7915fd-e76a-407d-825e-ae095f5d3c56\" (UID: \"8e7915fd-e76a-407d-825e-ae095f5d3c56\") "
Oct 30 22:48:47.940212 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:47.940114    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e7915fd-e76a-407d-825e-ae095f5d3c56-istio-token\" (UniqueName: \"kubernetes.io/projected/8e7915fd-e76a-407d-825e-ae095f5d3c56-istio-token\") pod \"8e7915fd-e76a-407d-825e-ae095f5d3c56\" (UID: \"8e7915fd-e76a-407d-825e-ae095f5d3c56\") "
Oct 30 22:48:47.940212 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:47.940174    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e7915fd-e76a-407d-825e-ae095f5d3c56-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e7915fd-e76a-407d-825e-ae095f5d3c56-credential-socket\") pod \"8e7915fd-e76a-407d-825e-ae095f5d3c56\" (UID: \"8e7915fd-e76a-407d-825e-ae095f5d3c56\") "
Oct 30 22:48:47.940328 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:47.940271    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e7915fd-e76a-407d-825e-ae095f5d3c56-kube-api-access-6vkj7\" (UniqueName: \"kubernetes.io/projected/8e7915fd-e76a-407d-825e-ae095f5d3c56-kube-api-access-6vkj7\") pod \"8e7915fd-e76a-407d-825e-ae095f5d3c56\" (UID: \"8e7915fd-e76a-407d-825e-ae095f5d3c56\") "
Oct 30 22:48:47.940393 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:47.940312    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e7915fd-e76a-407d-825e-ae095f5d3c56-workload-certs" pod "8e7915fd-e76a-407d-825e-ae095f5d3c56" (UID: "8e7915fd-e76a-407d-825e-ae095f5d3c56"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:48:47.940393 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:47.940340    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8e7915fd-e76a-407d-825e-ae095f5d3c56-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e7915fd-e76a-407d-825e-ae095f5d3c56-istio-podinfo\") pod \"8e7915fd-e76a-407d-825e-ae095f5d3c56\" (UID: \"8e7915fd-e76a-407d-825e-ae095f5d3c56\") "
Oct 30 22:48:47.940502 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:47.940432    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e7915fd-e76a-407d-825e-ae095f5d3c56-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e7915fd-e76a-407d-825e-ae095f5d3c56-istio-envoy\") pod \"8e7915fd-e76a-407d-825e-ae095f5d3c56\" (UID: \"8e7915fd-e76a-407d-825e-ae095f5d3c56\") "
Oct 30 22:48:47.940502 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:47.940482    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e7915fd-e76a-407d-825e-ae095f5d3c56-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e7915fd-e76a-407d-825e-ae095f5d3c56-workload-socket\") pod \"8e7915fd-e76a-407d-825e-ae095f5d3c56\" (UID: \"8e7915fd-e76a-407d-825e-ae095f5d3c56\") "
Oct 30 22:48:47.940783 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:47.940494    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e7915fd-e76a-407d-825e-ae095f5d3c56-credential-socket" pod "8e7915fd-e76a-407d-825e-ae095f5d3c56" (UID: "8e7915fd-e76a-407d-825e-ae095f5d3c56"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:48:47.940783 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:47.940568    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e7915fd-e76a-407d-825e-ae095f5d3c56-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e7915fd-e76a-407d-825e-ae095f5d3c56-istio-data\") pod \"8e7915fd-e76a-407d-825e-ae095f5d3c56\" (UID: \"8e7915fd-e76a-407d-825e-ae095f5d3c56\") "
Oct 30 22:48:47.940783 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:47.940706    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e7915fd-e76a-407d-825e-ae095f5d3c56-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e7915fd-e76a-407d-825e-ae095f5d3c56-istiod-ca-cert\") pod \"8e7915fd-e76a-407d-825e-ae095f5d3c56\" (UID: \"8e7915fd-e76a-407d-825e-ae095f5d3c56\") "
Oct 30 22:48:47.940783 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:47.940723    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e7915fd-e76a-407d-825e-ae095f5d3c56-workload-socket" pod "8e7915fd-e76a-407d-825e-ae095f5d3c56" (UID: "8e7915fd-e76a-407d-825e-ae095f5d3c56"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:48:47.941353 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:47.941320    4477 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1-workload-socket\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:48:47.941353 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:47.941351    4477 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1-istiod-ca-cert\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:48:47.941611 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:47.941366    4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9whg\" (UniqueName: \"kubernetes.io/projected/76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1-kube-api-access-d9whg\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:48:47.941611 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:47.941384    4477 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1-istio-envoy\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:48:47.941611 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:47.941398    4477 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e7915fd-e76a-407d-825e-ae095f5d3c56-workload-socket\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:48:47.941611 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:47.941410    4477 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1-istio-podinfo\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:48:47.941611 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:47.941427    4477 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e7915fd-e76a-407d-825e-ae095f5d3c56-workload-certs\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:48:47.941611 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:47.941440    4477 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e7915fd-e76a-407d-825e-ae095f5d3c56-credential-socket\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:48:47.941611 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:47.941451    4477 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1-istio-token\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:48:47.941611 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:47.941352    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e7915fd-e76a-407d-825e-ae095f5d3c56-istio-data" pod "8e7915fd-e76a-407d-825e-ae095f5d3c56" (UID: "8e7915fd-e76a-407d-825e-ae095f5d3c56"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:48:47.942030 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:47.941703    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e7915fd-e76a-407d-825e-ae095f5d3c56-istiod-ca-cert" pod "8e7915fd-e76a-407d-825e-ae095f5d3c56" (UID: "8e7915fd-e76a-407d-825e-ae095f5d3c56"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:48:47.942651 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:47.942629    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e7915fd-e76a-407d-825e-ae095f5d3c56-istio-token" pod "8e7915fd-e76a-407d-825e-ae095f5d3c56" (UID: "8e7915fd-e76a-407d-825e-ae095f5d3c56"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:48:47.942930 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:47.942914    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e7915fd-e76a-407d-825e-ae095f5d3c56-kube-api-access-6vkj7" pod "8e7915fd-e76a-407d-825e-ae095f5d3c56" (UID: "8e7915fd-e76a-407d-825e-ae095f5d3c56"). InnerVolumeSpecName "kube-api-access-6vkj7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:48:47.942980 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:47.942930    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e7915fd-e76a-407d-825e-ae095f5d3c56-istio-envoy" pod "8e7915fd-e76a-407d-825e-ae095f5d3c56" (UID: "8e7915fd-e76a-407d-825e-ae095f5d3c56"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:48:47.943586 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:47.943568    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8e7915fd-e76a-407d-825e-ae095f5d3c56-istio-podinfo" pod "8e7915fd-e76a-407d-825e-ae095f5d3c56" (UID: "8e7915fd-e76a-407d-825e-ae095f5d3c56"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:48:48.042624 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:48.042555    4477 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e7915fd-e76a-407d-825e-ae095f5d3c56-istio-data\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:48:48.042624 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:48.042579    4477 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e7915fd-e76a-407d-825e-ae095f5d3c56-istiod-ca-cert\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:48:48.042624 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:48.042588    4477 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e7915fd-e76a-407d-825e-ae095f5d3c56-istio-token\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:48:48.042624 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:48.042596    4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vkj7\" (UniqueName: \"kubernetes.io/projected/8e7915fd-e76a-407d-825e-ae095f5d3c56-kube-api-access-6vkj7\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:48:48.042624 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:48.042604    4477 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e7915fd-e76a-407d-825e-ae095f5d3c56-istio-podinfo\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:48:48.042624 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:48.042614    4477 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e7915fd-e76a-407d-825e-ae095f5d3c56-istio-envoy\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:48:48.474938 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:48.474912    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hq9vz"
Oct 30 22:48:48.475393 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:48.474949    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hq9vz" event={"ID":"76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1","Type":"ContainerDied","Data":"728439bf6521aeaca021a03e1dc4f70536707ee943306117437a01fca6c34d9c"}
Oct 30 22:48:48.475393 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:48.475014    4477 scope.go:122] "RemoveContainer" containerID="db69e1b6f0249022202674e28dc9b17682d88f0154d4960524be464b4e7046e4"
Oct 30 22:48:48.476527 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:48.476499    4477 generic.go:356] "Generic (PLEG): container finished" podID="8e7915fd-e76a-407d-825e-ae095f5d3c56" containerID="de5712116b50a298f8a609b6ca5f0b7b67659e2cfd6c70cb4060fe984f6318bc" exitCode=0
Oct 30 22:48:48.476656 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:48.476547    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-lg99w"
Oct 30 22:48:48.476656 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:48.476587    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-lg99w" event={"ID":"8e7915fd-e76a-407d-825e-ae095f5d3c56","Type":"ContainerDied","Data":"de5712116b50a298f8a609b6ca5f0b7b67659e2cfd6c70cb4060fe984f6318bc"}
Oct 30 22:48:48.476656 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:48.476640    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-lg99w" event={"ID":"8e7915fd-e76a-407d-825e-ae095f5d3c56","Type":"ContainerDied","Data":"58654c4afab4ae6b6d6811b7e4a553a3cd1ece0844548bbc8169da4d7090a23d"}
Oct 30 22:48:48.486966 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:48.486947    4477 scope.go:122] "RemoveContainer" containerID="de5712116b50a298f8a609b6ca5f0b7b67659e2cfd6c70cb4060fe984f6318bc"
Oct 30 22:48:48.496334 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:48.496314    4477 scope.go:122] "RemoveContainer" containerID="de5712116b50a298f8a609b6ca5f0b7b67659e2cfd6c70cb4060fe984f6318bc"
Oct 30 22:48:48.496600 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:48:48.496582    4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de5712116b50a298f8a609b6ca5f0b7b67659e2cfd6c70cb4060fe984f6318bc\": container with ID starting with de5712116b50a298f8a609b6ca5f0b7b67659e2cfd6c70cb4060fe984f6318bc not found: ID does not exist" containerID="de5712116b50a298f8a609b6ca5f0b7b67659e2cfd6c70cb4060fe984f6318bc"
Oct 30 22:48:48.496683 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:48.496612    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de5712116b50a298f8a609b6ca5f0b7b67659e2cfd6c70cb4060fe984f6318bc"} err="failed to get container status \"de5712116b50a298f8a609b6ca5f0b7b67659e2cfd6c70cb4060fe984f6318bc\": rpc error: code = NotFound desc = could not find container \"de5712116b50a298f8a609b6ca5f0b7b67659e2cfd6c70cb4060fe984f6318bc\": container with ID starting with de5712116b50a298f8a609b6ca5f0b7b67659e2cfd6c70cb4060fe984f6318bc not found: ID does not exist"
Oct 30 22:48:48.537715 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:48.537690    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-lg99w"]
Oct 30 22:48:48.548653 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:48.548629    4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-012ba0-openshift-default-846bcc5bfc-lg99w"]
Oct 30 22:48:48.574377 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:48.574354    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hq9vz"]
Oct 30 22:48:48.577686 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:48.577663    4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-0ad7c5-openshift-default-788b484d5c-hq9vz"]
Oct 30 22:48:50.193734 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:50.193701    4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1" path="/var/lib/kubelet/pods/76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1/volumes"
Oct 30 22:48:50.194200 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:50.194185    4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e7915fd-e76a-407d-825e-ae095f5d3c56" path="/var/lib/kubelet/pods/8e7915fd-e76a-407d-825e-ae095f5d3c56/volumes"
Oct 30 22:48:54.279033 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:54.278990    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c13db0-749698df58-xzmc9"]
Oct 30 22:48:54.279522 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:54.279245    4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c13db0-749698df58-xzmc9" podUID="25d06fe7-35c8-464e-9f50-1cf18a613fa1" containerName="maas-api" containerID="cri-o://3af11adf035bd793840717919bab7b231df6b494c000884f8a83df685e3414a5" gracePeriod=30
Oct 30 22:48:54.583021 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:54.581673    4477 generic.go:356] "Generic (PLEG): container finished" podID="25d06fe7-35c8-464e-9f50-1cf18a613fa1" containerID="3af11adf035bd793840717919bab7b231df6b494c000884f8a83df685e3414a5" exitCode=0
Oct 30 22:48:54.583021 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:54.581851    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c13db0-749698df58-xzmc9" event={"ID":"25d06fe7-35c8-464e-9f50-1cf18a613fa1","Type":"ContainerDied","Data":"3af11adf035bd793840717919bab7b231df6b494c000884f8a83df685e3414a5"}
Oct 30 22:48:54.640504 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:54.640476    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c13db0-749698df58-xzmc9"
Oct 30 22:48:54.715922 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:54.715867    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25d06fe7-35c8-464e-9f50-1cf18a613fa1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25d06fe7-35c8-464e-9f50-1cf18a613fa1-openshift-service-ca\") pod \"25d06fe7-35c8-464e-9f50-1cf18a613fa1\" (UID: \"25d06fe7-35c8-464e-9f50-1cf18a613fa1\") "
Oct 30 22:48:54.716112 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:54.716092    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25d06fe7-35c8-464e-9f50-1cf18a613fa1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25d06fe7-35c8-464e-9f50-1cf18a613fa1-maas-api-tls\") pod \"25d06fe7-35c8-464e-9f50-1cf18a613fa1\" (UID: \"25d06fe7-35c8-464e-9f50-1cf18a613fa1\") "
Oct 30 22:48:54.716273 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:54.716246    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25d06fe7-35c8-464e-9f50-1cf18a613fa1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/25d06fe7-35c8-464e-9f50-1cf18a613fa1-metrics-certs\") pod \"25d06fe7-35c8-464e-9f50-1cf18a613fa1\" (UID: \"25d06fe7-35c8-464e-9f50-1cf18a613fa1\") "
Oct 30 22:48:54.716339 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:54.716301    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25d06fe7-35c8-464e-9f50-1cf18a613fa1-kube-api-access-n6b5t\" (UniqueName: \"kubernetes.io/projected/25d06fe7-35c8-464e-9f50-1cf18a613fa1-kube-api-access-n6b5t\") pod \"25d06fe7-35c8-464e-9f50-1cf18a613fa1\" (UID: \"25d06fe7-35c8-464e-9f50-1cf18a613fa1\") "
Oct 30 22:48:54.716752 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:54.716726    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25d06fe7-35c8-464e-9f50-1cf18a613fa1-openshift-service-ca" pod "25d06fe7-35c8-464e-9f50-1cf18a613fa1" (UID: "25d06fe7-35c8-464e-9f50-1cf18a613fa1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:48:54.720809 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:54.720778    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25d06fe7-35c8-464e-9f50-1cf18a613fa1-maas-api-tls" pod "25d06fe7-35c8-464e-9f50-1cf18a613fa1" (UID: "25d06fe7-35c8-464e-9f50-1cf18a613fa1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:48:54.743271 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:54.736187    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25d06fe7-35c8-464e-9f50-1cf18a613fa1-metrics-certs" pod "25d06fe7-35c8-464e-9f50-1cf18a613fa1" (UID: "25d06fe7-35c8-464e-9f50-1cf18a613fa1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:48:54.743271 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:54.739138    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25d06fe7-35c8-464e-9f50-1cf18a613fa1-kube-api-access-n6b5t" pod "25d06fe7-35c8-464e-9f50-1cf18a613fa1" (UID: "25d06fe7-35c8-464e-9f50-1cf18a613fa1"). InnerVolumeSpecName "kube-api-access-n6b5t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:48:54.818998 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:54.818960    4477 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/25d06fe7-35c8-464e-9f50-1cf18a613fa1-maas-api-tls\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:48:54.818998 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:54.818996    4477 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/25d06fe7-35c8-464e-9f50-1cf18a613fa1-metrics-certs\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:48:54.819268 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:54.819011    4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6b5t\" (UniqueName: \"kubernetes.io/projected/25d06fe7-35c8-464e-9f50-1cf18a613fa1-kube-api-access-n6b5t\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:48:54.819268 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:54.819027    4477 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/25d06fe7-35c8-464e-9f50-1cf18a613fa1-openshift-service-ca\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:48:55.590558 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:55.590525    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c13db0-749698df58-xzmc9" event={"ID":"25d06fe7-35c8-464e-9f50-1cf18a613fa1","Type":"ContainerDied","Data":"931d4a3105cee7b42a522c3af67bc0ba623c16b60601a3d852818a94b8f9d1e1"}
Oct 30 22:48:55.590558 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:55.590541    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c13db0-749698df58-xzmc9"
Oct 30 22:48:55.591148 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:55.590585    4477 scope.go:122] "RemoveContainer" containerID="3af11adf035bd793840717919bab7b231df6b494c000884f8a83df685e3414a5"
Oct 30 22:48:55.643303 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:55.643271    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c13db0-749698df58-xzmc9"]
Oct 30 22:48:55.658926 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:55.658893    4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c13db0-749698df58-xzmc9"]
Oct 30 22:48:56.193642 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:56.193612    4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25d06fe7-35c8-464e-9f50-1cf18a613fa1" path="/var/lib/kubelet/pods/25d06fe7-35c8-464e-9f50-1cf18a613fa1/volumes"
Oct 30 22:48:56.266553 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:48:56.266502    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 22:48:57.162595 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:48:57.162563    4477 scope.go:122] "RemoveContainer" containerID="3b091c37446e46d81a03d352931431ae28cf96b5997c11435a783b49bc2ddee4"
Oct 30 22:49:00.258981 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:00.258949    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6e0edc-6fb6b889ff-96k9w"]
Oct 30 22:49:00.259472 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:00.259276    4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6e0edc-6fb6b889ff-96k9w" podUID="d6d933b6-d604-4487-aad9-41feae842796" containerName="maas-api" containerID="cri-o://0216c55d852e529e1b352b20ca4a09ae2d0e38faf635b5152bf661dee93275c9" gracePeriod=30
Oct 30 22:49:00.620865 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:00.620830    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6e0edc-6fb6b889ff-96k9w"
Oct 30 22:49:00.622620 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:00.622590    4477 generic.go:356] "Generic (PLEG): container finished" podID="d6d933b6-d604-4487-aad9-41feae842796" containerID="0216c55d852e529e1b352b20ca4a09ae2d0e38faf635b5152bf661dee93275c9" exitCode=0
Oct 30 22:49:00.622763 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:00.622629    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6e0edc-6fb6b889ff-96k9w"
Oct 30 22:49:00.622763 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:00.622670    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6e0edc-6fb6b889ff-96k9w" event={"ID":"d6d933b6-d604-4487-aad9-41feae842796","Type":"ContainerDied","Data":"0216c55d852e529e1b352b20ca4a09ae2d0e38faf635b5152bf661dee93275c9"}
Oct 30 22:49:00.622763 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:00.622727    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6e0edc-6fb6b889ff-96k9w" event={"ID":"d6d933b6-d604-4487-aad9-41feae842796","Type":"ContainerDied","Data":"1bee247c54e880cf4e0081ef0b05cd7461715ef66fdd085f914a566fded3a17a"}
Oct 30 22:49:00.622763 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:00.622755    4477 scope.go:122] "RemoveContainer" containerID="0216c55d852e529e1b352b20ca4a09ae2d0e38faf635b5152bf661dee93275c9"
Oct 30 22:49:00.639425 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:00.638329    4477 scope.go:122] "RemoveContainer" containerID="0216c55d852e529e1b352b20ca4a09ae2d0e38faf635b5152bf661dee93275c9"
Oct 30 22:49:00.639425 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:49:00.638651    4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0216c55d852e529e1b352b20ca4a09ae2d0e38faf635b5152bf661dee93275c9\": container with ID starting with 0216c55d852e529e1b352b20ca4a09ae2d0e38faf635b5152bf661dee93275c9 not found: ID does not exist" containerID="0216c55d852e529e1b352b20ca4a09ae2d0e38faf635b5152bf661dee93275c9"
Oct 30 22:49:00.639425 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:00.638689    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0216c55d852e529e1b352b20ca4a09ae2d0e38faf635b5152bf661dee93275c9"} err="failed to get container status \"0216c55d852e529e1b352b20ca4a09ae2d0e38faf635b5152bf661dee93275c9\": rpc error: code = NotFound desc = could not find container \"0216c55d852e529e1b352b20ca4a09ae2d0e38faf635b5152bf661dee93275c9\": container with ID starting with 0216c55d852e529e1b352b20ca4a09ae2d0e38faf635b5152bf661dee93275c9 not found: ID does not exist"
Oct 30 22:49:00.702172 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:00.702137    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6d933b6-d604-4487-aad9-41feae842796-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6d933b6-d604-4487-aad9-41feae842796-openshift-service-ca\") pod \"d6d933b6-d604-4487-aad9-41feae842796\" (UID: \"d6d933b6-d604-4487-aad9-41feae842796\") "
Oct 30 22:49:00.702628 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:00.702606    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6d933b6-d604-4487-aad9-41feae842796-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6d933b6-d604-4487-aad9-41feae842796-maas-api-tls\") pod \"d6d933b6-d604-4487-aad9-41feae842796\" (UID: \"d6d933b6-d604-4487-aad9-41feae842796\") "
Oct 30 22:49:00.702775 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:00.702761    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6d933b6-d604-4487-aad9-41feae842796-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6d933b6-d604-4487-aad9-41feae842796-metrics-certs\") pod \"d6d933b6-d604-4487-aad9-41feae842796\" (UID: \"d6d933b6-d604-4487-aad9-41feae842796\") "
Oct 30 22:49:00.702903 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:00.702889    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6d933b6-d604-4487-aad9-41feae842796-kube-api-access-nzqx9\" (UniqueName: \"kubernetes.io/projected/d6d933b6-d604-4487-aad9-41feae842796-kube-api-access-nzqx9\") pod \"d6d933b6-d604-4487-aad9-41feae842796\" (UID: \"d6d933b6-d604-4487-aad9-41feae842796\") "
Oct 30 22:49:00.711079 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:00.709494    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6d933b6-d604-4487-aad9-41feae842796-openshift-service-ca" pod "d6d933b6-d604-4487-aad9-41feae842796" (UID: "d6d933b6-d604-4487-aad9-41feae842796"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:49:00.725670 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:00.725630    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6d933b6-d604-4487-aad9-41feae842796-kube-api-access-nzqx9" pod "d6d933b6-d604-4487-aad9-41feae842796" (UID: "d6d933b6-d604-4487-aad9-41feae842796"). InnerVolumeSpecName "kube-api-access-nzqx9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:49:00.726355 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:00.726308    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6d933b6-d604-4487-aad9-41feae842796-metrics-certs" pod "d6d933b6-d604-4487-aad9-41feae842796" (UID: "d6d933b6-d604-4487-aad9-41feae842796"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:49:00.728223 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:00.728179    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6d933b6-d604-4487-aad9-41feae842796-maas-api-tls" pod "d6d933b6-d604-4487-aad9-41feae842796" (UID: "d6d933b6-d604-4487-aad9-41feae842796"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:49:00.805253 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:00.805216    4477 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6d933b6-d604-4487-aad9-41feae842796-maas-api-tls\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:49:00.805253 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:00.805254    4477 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6d933b6-d604-4487-aad9-41feae842796-metrics-certs\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:49:00.805518 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:00.805269    4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzqx9\" (UniqueName: \"kubernetes.io/projected/d6d933b6-d604-4487-aad9-41feae842796-kube-api-access-nzqx9\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:49:00.805518 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:00.805304    4477 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6d933b6-d604-4487-aad9-41feae842796-openshift-service-ca\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:49:00.968248 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:00.968214    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6e0edc-6fb6b889ff-96k9w"]
Oct 30 22:49:00.973797 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:00.973773    4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6e0edc-6fb6b889ff-96k9w"]
Oct 30 22:49:02.194448 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:02.194416    4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6d933b6-d604-4487-aad9-41feae842796" path="/var/lib/kubelet/pods/d6d933b6-d604-4487-aad9-41feae842796/volumes"
Oct 30 22:49:08.415225 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:08.415187    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-q7w2l"]
Oct 30 22:49:08.415819 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:08.415439    4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-q7w2l" podUID="39eb3b4c-a5b3-4274-86d0-335fb7851d4a" containerName="istio-proxy" containerID="cri-o://47040e2bf8771f24efbf41faedee8e3d4693887b207bce53f01dc36107c13db2" gracePeriod=30
Oct 30 22:49:11.893238 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:11.893201    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-867f12/test-model-867f12-kserve-6694d95cf9-jcxgf"]
Oct 30 22:49:13.687704 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:13.687644    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-q7w2l"
Oct 30 22:49:13.706269 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:13.706239    4477 generic.go:356] "Generic (PLEG): container finished" podID="39eb3b4c-a5b3-4274-86d0-335fb7851d4a" containerID="47040e2bf8771f24efbf41faedee8e3d4693887b207bce53f01dc36107c13db2" exitCode=0
Oct 30 22:49:13.706401 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:13.706287    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-q7w2l"
Oct 30 22:49:13.706401 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:13.706325    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-q7w2l" event={"ID":"39eb3b4c-a5b3-4274-86d0-335fb7851d4a","Type":"ContainerDied","Data":"47040e2bf8771f24efbf41faedee8e3d4693887b207bce53f01dc36107c13db2"}
Oct 30 22:49:13.706401 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:13.706370    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-q7w2l" event={"ID":"39eb3b4c-a5b3-4274-86d0-335fb7851d4a","Type":"ContainerDied","Data":"19eba94abb4aecc70c1bf1c86c5927c9ffc80fda4ff159ae13f25319d5fcff67"}
Oct 30 22:49:13.706401 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:13.706398    4477 scope.go:122] "RemoveContainer" containerID="47040e2bf8771f24efbf41faedee8e3d4693887b207bce53f01dc36107c13db2"
Oct 30 22:49:13.718583 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:13.718561    4477 scope.go:122] "RemoveContainer" containerID="47040e2bf8771f24efbf41faedee8e3d4693887b207bce53f01dc36107c13db2"
Oct 30 22:49:13.718970 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:49:13.718951    4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47040e2bf8771f24efbf41faedee8e3d4693887b207bce53f01dc36107c13db2\": container with ID starting with 47040e2bf8771f24efbf41faedee8e3d4693887b207bce53f01dc36107c13db2 not found: ID does not exist" containerID="47040e2bf8771f24efbf41faedee8e3d4693887b207bce53f01dc36107c13db2"
Oct 30 22:49:13.719080 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:13.718981    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47040e2bf8771f24efbf41faedee8e3d4693887b207bce53f01dc36107c13db2"} err="failed to get container status \"47040e2bf8771f24efbf41faedee8e3d4693887b207bce53f01dc36107c13db2\": rpc error: code = NotFound desc = could not find container \"47040e2bf8771f24efbf41faedee8e3d4693887b207bce53f01dc36107c13db2\": container with ID starting with 47040e2bf8771f24efbf41faedee8e3d4693887b207bce53f01dc36107c13db2 not found: ID does not exist"
Oct 30 22:49:13.779508 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:13.779479    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39eb3b4c-a5b3-4274-86d0-335fb7851d4a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39eb3b4c-a5b3-4274-86d0-335fb7851d4a-istio-data\") pod \"39eb3b4c-a5b3-4274-86d0-335fb7851d4a\" (UID: \"39eb3b4c-a5b3-4274-86d0-335fb7851d4a\") "
Oct 30 22:49:13.779692 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:13.779519    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39eb3b4c-a5b3-4274-86d0-335fb7851d4a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39eb3b4c-a5b3-4274-86d0-335fb7851d4a-workload-certs\") pod \"39eb3b4c-a5b3-4274-86d0-335fb7851d4a\" (UID: \"39eb3b4c-a5b3-4274-86d0-335fb7851d4a\") "
Oct 30 22:49:13.779692 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:13.779553    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39eb3b4c-a5b3-4274-86d0-335fb7851d4a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39eb3b4c-a5b3-4274-86d0-335fb7851d4a-istiod-ca-cert\") pod \"39eb3b4c-a5b3-4274-86d0-335fb7851d4a\" (UID: \"39eb3b4c-a5b3-4274-86d0-335fb7851d4a\") "
Oct 30 22:49:13.779692 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:13.779590    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/39eb3b4c-a5b3-4274-86d0-335fb7851d4a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39eb3b4c-a5b3-4274-86d0-335fb7851d4a-istio-podinfo\") pod \"39eb3b4c-a5b3-4274-86d0-335fb7851d4a\" (UID: \"39eb3b4c-a5b3-4274-86d0-335fb7851d4a\") "
Oct 30 22:49:13.779692 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:13.779629    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39eb3b4c-a5b3-4274-86d0-335fb7851d4a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39eb3b4c-a5b3-4274-86d0-335fb7851d4a-istio-envoy\") pod \"39eb3b4c-a5b3-4274-86d0-335fb7851d4a\" (UID: \"39eb3b4c-a5b3-4274-86d0-335fb7851d4a\") "
Oct 30 22:49:13.779692 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:13.779682    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39eb3b4c-a5b3-4274-86d0-335fb7851d4a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39eb3b4c-a5b3-4274-86d0-335fb7851d4a-credential-socket\") pod \"39eb3b4c-a5b3-4274-86d0-335fb7851d4a\" (UID: \"39eb3b4c-a5b3-4274-86d0-335fb7851d4a\") "
Oct 30 22:49:13.779997 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:13.779721    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39eb3b4c-a5b3-4274-86d0-335fb7851d4a-istio-token\" (UniqueName: \"kubernetes.io/projected/39eb3b4c-a5b3-4274-86d0-335fb7851d4a-istio-token\") pod \"39eb3b4c-a5b3-4274-86d0-335fb7851d4a\" (UID: \"39eb3b4c-a5b3-4274-86d0-335fb7851d4a\") "
Oct 30 22:49:13.779997 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:13.779756    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39eb3b4c-a5b3-4274-86d0-335fb7851d4a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39eb3b4c-a5b3-4274-86d0-335fb7851d4a-workload-socket\") pod \"39eb3b4c-a5b3-4274-86d0-335fb7851d4a\" (UID: \"39eb3b4c-a5b3-4274-86d0-335fb7851d4a\") "
Oct 30 22:49:13.779997 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:13.779774    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39eb3b4c-a5b3-4274-86d0-335fb7851d4a-istio-data" pod "39eb3b4c-a5b3-4274-86d0-335fb7851d4a" (UID: "39eb3b4c-a5b3-4274-86d0-335fb7851d4a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:49:13.779997 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:13.779802    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39eb3b4c-a5b3-4274-86d0-335fb7851d4a-kube-api-access-645tk\" (UniqueName: \"kubernetes.io/projected/39eb3b4c-a5b3-4274-86d0-335fb7851d4a-kube-api-access-645tk\") pod \"39eb3b4c-a5b3-4274-86d0-335fb7851d4a\" (UID: \"39eb3b4c-a5b3-4274-86d0-335fb7851d4a\") "
Oct 30 22:49:13.780270 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:13.780241    4477 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39eb3b4c-a5b3-4274-86d0-335fb7851d4a-istio-data\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:49:13.782066 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:13.780612    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39eb3b4c-a5b3-4274-86d0-335fb7851d4a-credential-socket" pod "39eb3b4c-a5b3-4274-86d0-335fb7851d4a" (UID: "39eb3b4c-a5b3-4274-86d0-335fb7851d4a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:49:13.782066 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:13.780797    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39eb3b4c-a5b3-4274-86d0-335fb7851d4a-workload-certs" pod "39eb3b4c-a5b3-4274-86d0-335fb7851d4a" (UID: "39eb3b4c-a5b3-4274-86d0-335fb7851d4a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:49:13.782066 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:13.780809    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39eb3b4c-a5b3-4274-86d0-335fb7851d4a-workload-socket" pod "39eb3b4c-a5b3-4274-86d0-335fb7851d4a" (UID: "39eb3b4c-a5b3-4274-86d0-335fb7851d4a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:49:13.782066 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:13.780907    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39eb3b4c-a5b3-4274-86d0-335fb7851d4a-istiod-ca-cert" pod "39eb3b4c-a5b3-4274-86d0-335fb7851d4a" (UID: "39eb3b4c-a5b3-4274-86d0-335fb7851d4a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:49:13.782518 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:13.782480    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39eb3b4c-a5b3-4274-86d0-335fb7851d4a-istio-envoy" pod "39eb3b4c-a5b3-4274-86d0-335fb7851d4a" (UID: "39eb3b4c-a5b3-4274-86d0-335fb7851d4a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:49:13.782649 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:13.782620    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39eb3b4c-a5b3-4274-86d0-335fb7851d4a-kube-api-access-645tk" pod "39eb3b4c-a5b3-4274-86d0-335fb7851d4a" (UID: "39eb3b4c-a5b3-4274-86d0-335fb7851d4a"). InnerVolumeSpecName "kube-api-access-645tk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:49:13.783007 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:13.782982    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39eb3b4c-a5b3-4274-86d0-335fb7851d4a-istio-token" pod "39eb3b4c-a5b3-4274-86d0-335fb7851d4a" (UID: "39eb3b4c-a5b3-4274-86d0-335fb7851d4a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:49:13.784119 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:13.784096    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/39eb3b4c-a5b3-4274-86d0-335fb7851d4a-istio-podinfo" pod "39eb3b4c-a5b3-4274-86d0-335fb7851d4a" (UID: "39eb3b4c-a5b3-4274-86d0-335fb7851d4a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:49:13.881907 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:13.881880    4477 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39eb3b4c-a5b3-4274-86d0-335fb7851d4a-istio-token\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:49:13.881907 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:13.881909    4477 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39eb3b4c-a5b3-4274-86d0-335fb7851d4a-workload-socket\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:49:13.882141 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:13.881920    4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-645tk\" (UniqueName: \"kubernetes.io/projected/39eb3b4c-a5b3-4274-86d0-335fb7851d4a-kube-api-access-645tk\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:49:13.882141 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:13.881930    4477 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39eb3b4c-a5b3-4274-86d0-335fb7851d4a-workload-certs\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:49:13.882141 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:13.881939    4477 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39eb3b4c-a5b3-4274-86d0-335fb7851d4a-istiod-ca-cert\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:49:13.882141 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:13.881946    4477 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39eb3b4c-a5b3-4274-86d0-335fb7851d4a-istio-podinfo\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:49:13.882141 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:13.881958    4477 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39eb3b4c-a5b3-4274-86d0-335fb7851d4a-istio-envoy\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:49:13.882141 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:13.881968    4477 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39eb3b4c-a5b3-4274-86d0-335fb7851d4a-credential-socket\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:49:14.051631 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:14.051597    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-q7w2l"]
Oct 30 22:49:14.062629 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:14.062593    4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-6e0edc-openshift-default-7968b4dddf-q7w2l"]
Oct 30 22:49:14.193490 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:14.193462    4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39eb3b4c-a5b3-4274-86d0-335fb7851d4a" path="/var/lib/kubelet/pods/39eb3b4c-a5b3-4274-86d0-335fb7851d4a/volumes"
Oct 30 22:49:15.347972 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:15.347945    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w0-5983ea/auth-test-model-5983ea-kserve-d4df66dd5-96zv4"]
Oct 30 22:49:15.348564 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:15.348548    4477 cpu_manager.go:414] "containerMap: removing container" podUID="39eb3b4c-a5b3-4274-86d0-335fb7851d4a" containerName="istio-proxy"
Oct 30 22:49:15.348609 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:15.348565    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39eb3b4c-a5b3-4274-86d0-335fb7851d4a" containerName="istio-proxy"
Oct 30 22:49:15.348609 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:15.348588    4477 cpu_manager.go:414] "containerMap: removing container" podUID="8e7915fd-e76a-407d-825e-ae095f5d3c56" containerName="istio-proxy"
Oct 30 22:49:15.348609 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:15.348593    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e7915fd-e76a-407d-825e-ae095f5d3c56" containerName="istio-proxy"
Oct 30 22:49:15.348609 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:15.348604    4477 cpu_manager.go:414] "containerMap: removing container" podUID="616d47a3-a660-48d3-9714-9e0f45a5660a" containerName="maas-api"
Oct 30 22:49:15.348743 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:15.348609    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="616d47a3-a660-48d3-9714-9e0f45a5660a" containerName="maas-api"
Oct 30 22:49:15.348743 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:15.348627    4477 cpu_manager.go:414] "containerMap: removing container" podUID="d6d933b6-d604-4487-aad9-41feae842796" containerName="maas-api"
Oct 30 22:49:15.348743 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:15.348632    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6d933b6-d604-4487-aad9-41feae842796" containerName="maas-api"
Oct 30 22:49:15.348743 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:15.348644    4477 cpu_manager.go:414] "containerMap: removing container" podUID="8c927c80-1da0-4b5b-8655-c67e8a35f000" containerName="maas-api"
Oct 30 22:49:15.348743 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:15.348649    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c927c80-1da0-4b5b-8655-c67e8a35f000" containerName="maas-api"
Oct 30 22:49:15.348743 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:15.348660    4477 cpu_manager.go:414] "containerMap: removing container" podUID="bedea217-88bb-4871-9f6f-0eb57c9b0654" containerName="main"
Oct 30 22:49:15.348743 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:15.348664    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bedea217-88bb-4871-9f6f-0eb57c9b0654" containerName="main"
Oct 30 22:49:15.348743 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:15.348677    4477 cpu_manager.go:414] "containerMap: removing container" podUID="76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1" containerName="istio-proxy"
Oct 30 22:49:15.348743 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:15.348681    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1" containerName="istio-proxy"
Oct 30 22:49:15.348743 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:15.348687    4477 cpu_manager.go:414] "containerMap: removing container" podUID="13dfee80-6a6d-4b97-be9d-728a85a59e5b" containerName="main"
Oct 30 22:49:15.348743 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:15.348692    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13dfee80-6a6d-4b97-be9d-728a85a59e5b" containerName="main"
Oct 30 22:49:15.348743 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:15.348702    4477 cpu_manager.go:414] "containerMap: removing container" podUID="25d06fe7-35c8-464e-9f50-1cf18a613fa1" containerName="maas-api"
Oct 30 22:49:15.348743 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:15.348706    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25d06fe7-35c8-464e-9f50-1cf18a613fa1" containerName="maas-api"
Oct 30 22:49:15.348743 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:15.348714    4477 cpu_manager.go:414] "containerMap: removing container" podUID="b47ab287-9d84-468d-8afb-28b3f6b862e2" containerName="maas-api"
Oct 30 22:49:15.348743 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:15.348718    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b47ab287-9d84-468d-8afb-28b3f6b862e2" containerName="maas-api"
Oct 30 22:49:15.349206 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:15.348819    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c927c80-1da0-4b5b-8655-c67e8a35f000" containerName="maas-api"
Oct 30 22:49:15.349206 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:15.348828    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="bedea217-88bb-4871-9f6f-0eb57c9b0654" containerName="main"
Oct 30 22:49:15.349206 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:15.348835    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="76fb88a8-b2ba-42c2-99c2-89fb9fbd4ca1" containerName="istio-proxy"
Oct 30 22:49:15.349206 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:15.348842    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="13dfee80-6a6d-4b97-be9d-728a85a59e5b" containerName="main"
Oct 30 22:49:15.349206 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:15.348848    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="25d06fe7-35c8-464e-9f50-1cf18a613fa1" containerName="maas-api"
Oct 30 22:49:15.349206 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:15.348853    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="b47ab287-9d84-468d-8afb-28b3f6b862e2" containerName="maas-api"
Oct 30 22:49:15.349206 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:15.348859    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="39eb3b4c-a5b3-4274-86d0-335fb7851d4a" containerName="istio-proxy"
Oct 30 22:49:15.349206 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:15.348867    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e7915fd-e76a-407d-825e-ae095f5d3c56" containerName="istio-proxy"
Oct 30 22:49:15.349206 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:15.348873    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="616d47a3-a660-48d3-9714-9e0f45a5660a" containerName="maas-api"
Oct 30 22:49:15.349206 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:15.348879    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6d933b6-d604-4487-aad9-41feae842796" containerName="maas-api"
Oct 30 22:49:15.353650 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:15.353629    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-5983ea/auth-test-model-5983ea-kserve-d4df66dd5-96zv4"
Oct 30 22:49:15.374758 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:15.374729    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-5983ea/auth-test-model-5983ea-kserve-d4df66dd5-96zv4"]
Oct 30 22:49:15.382402 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:15.382377    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-5983ea\"/\"auth-test-model-5983ea-kserve-self-signed-certs\""
Oct 30 22:49:15.382571 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:15.382450    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-5983ea\"/\"default-dockercfg-d8gnh\""
Oct 30 22:49:15.386242 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:15.386190    4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-5983ea\"/\"kube-root-ca.crt\""
Oct 30 22:49:15.386377 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:15.386290    4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-5983ea\"/\"openshift-service-ca.crt\""
Oct 30 22:49:15.502512 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:15.502482    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e727c25-f834-4b00-aec6-9f3c7cb0b97b-dshm\") pod \"auth-test-model-5983ea-kserve-d4df66dd5-96zv4\" (UID: \"1e727c25-f834-4b00-aec6-9f3c7cb0b97b\") " pod="ai-tenant-e2e-shared-a-w0-5983ea/auth-test-model-5983ea-kserve-d4df66dd5-96zv4"
Oct 30 22:49:15.502512 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:15.502515    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e727c25-f834-4b00-aec6-9f3c7cb0b97b-tmp-dir\") pod \"auth-test-model-5983ea-kserve-d4df66dd5-96zv4\" (UID: \"1e727c25-f834-4b00-aec6-9f3c7cb0b97b\") " pod="ai-tenant-e2e-shared-a-w0-5983ea/auth-test-model-5983ea-kserve-d4df66dd5-96zv4"
Oct 30 22:49:15.502784 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:15.502560    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbjmg\" (UniqueName: \"kubernetes.io/projected/1e727c25-f834-4b00-aec6-9f3c7cb0b97b-kube-api-access-hbjmg\") pod \"auth-test-model-5983ea-kserve-d4df66dd5-96zv4\" (UID: \"1e727c25-f834-4b00-aec6-9f3c7cb0b97b\") " pod="ai-tenant-e2e-shared-a-w0-5983ea/auth-test-model-5983ea-kserve-d4df66dd5-96zv4"
Oct 30 22:49:15.502784 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:15.502680    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e727c25-f834-4b00-aec6-9f3c7cb0b97b-tls-certs\") pod \"auth-test-model-5983ea-kserve-d4df66dd5-96zv4\" (UID: \"1e727c25-f834-4b00-aec6-9f3c7cb0b97b\") " pod="ai-tenant-e2e-shared-a-w0-5983ea/auth-test-model-5983ea-kserve-d4df66dd5-96zv4"
Oct 30 22:49:15.502784 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:15.502747    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e727c25-f834-4b00-aec6-9f3c7cb0b97b-model-cache\") pod \"auth-test-model-5983ea-kserve-d4df66dd5-96zv4\" (UID: \"1e727c25-f834-4b00-aec6-9f3c7cb0b97b\") " pod="ai-tenant-e2e-shared-a-w0-5983ea/auth-test-model-5983ea-kserve-d4df66dd5-96zv4"
Oct 30 22:49:15.502957 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:15.502875    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e727c25-f834-4b00-aec6-9f3c7cb0b97b-home\") pod \"auth-test-model-5983ea-kserve-d4df66dd5-96zv4\" (UID: \"1e727c25-f834-4b00-aec6-9f3c7cb0b97b\") " pod="ai-tenant-e2e-shared-a-w0-5983ea/auth-test-model-5983ea-kserve-d4df66dd5-96zv4"
Oct 30 22:49:15.604857 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:15.604761    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e727c25-f834-4b00-aec6-9f3c7cb0b97b-model-cache\") pod \"auth-test-model-5983ea-kserve-d4df66dd5-96zv4\" (UID: \"1e727c25-f834-4b00-aec6-9f3c7cb0b97b\") " pod="ai-tenant-e2e-shared-a-w0-5983ea/auth-test-model-5983ea-kserve-d4df66dd5-96zv4"
Oct 30 22:49:15.605034 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:15.604922    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e727c25-f834-4b00-aec6-9f3c7cb0b97b-home\") pod \"auth-test-model-5983ea-kserve-d4df66dd5-96zv4\" (UID: \"1e727c25-f834-4b00-aec6-9f3c7cb0b97b\") " pod="ai-tenant-e2e-shared-a-w0-5983ea/auth-test-model-5983ea-kserve-d4df66dd5-96zv4"
Oct 30 22:49:15.605034 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:15.605024    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e727c25-f834-4b00-aec6-9f3c7cb0b97b-dshm\") pod \"auth-test-model-5983ea-kserve-d4df66dd5-96zv4\" (UID: \"1e727c25-f834-4b00-aec6-9f3c7cb0b97b\") " pod="ai-tenant-e2e-shared-a-w0-5983ea/auth-test-model-5983ea-kserve-d4df66dd5-96zv4"
Oct 30 22:49:15.605180 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:15.605104    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e727c25-f834-4b00-aec6-9f3c7cb0b97b-tmp-dir\") pod \"auth-test-model-5983ea-kserve-d4df66dd5-96zv4\" (UID: \"1e727c25-f834-4b00-aec6-9f3c7cb0b97b\") " pod="ai-tenant-e2e-shared-a-w0-5983ea/auth-test-model-5983ea-kserve-d4df66dd5-96zv4"
Oct 30 22:49:15.605242 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:15.605189    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbjmg\" (UniqueName: \"kubernetes.io/projected/1e727c25-f834-4b00-aec6-9f3c7cb0b97b-kube-api-access-hbjmg\") pod \"auth-test-model-5983ea-kserve-d4df66dd5-96zv4\" (UID: \"1e727c25-f834-4b00-aec6-9f3c7cb0b97b\") " pod="ai-tenant-e2e-shared-a-w0-5983ea/auth-test-model-5983ea-kserve-d4df66dd5-96zv4"
Oct 30 22:49:15.605299 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:15.605234    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e727c25-f834-4b00-aec6-9f3c7cb0b97b-tls-certs\") pod \"auth-test-model-5983ea-kserve-d4df66dd5-96zv4\" (UID: \"1e727c25-f834-4b00-aec6-9f3c7cb0b97b\") " pod="ai-tenant-e2e-shared-a-w0-5983ea/auth-test-model-5983ea-kserve-d4df66dd5-96zv4"
Oct 30 22:49:15.605557 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:15.605267    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e727c25-f834-4b00-aec6-9f3c7cb0b97b-model-cache\") pod \"auth-test-model-5983ea-kserve-d4df66dd5-96zv4\" (UID: \"1e727c25-f834-4b00-aec6-9f3c7cb0b97b\") " pod="ai-tenant-e2e-shared-a-w0-5983ea/auth-test-model-5983ea-kserve-d4df66dd5-96zv4"
Oct 30 22:49:15.605851 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:15.605828    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e727c25-f834-4b00-aec6-9f3c7cb0b97b-home\") pod \"auth-test-model-5983ea-kserve-d4df66dd5-96zv4\" (UID: \"1e727c25-f834-4b00-aec6-9f3c7cb0b97b\") " pod="ai-tenant-e2e-shared-a-w0-5983ea/auth-test-model-5983ea-kserve-d4df66dd5-96zv4"
Oct 30 22:49:15.606110 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:15.606083    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e727c25-f834-4b00-aec6-9f3c7cb0b97b-tmp-dir\") pod \"auth-test-model-5983ea-kserve-d4df66dd5-96zv4\" (UID: \"1e727c25-f834-4b00-aec6-9f3c7cb0b97b\") " pod="ai-tenant-e2e-shared-a-w0-5983ea/auth-test-model-5983ea-kserve-d4df66dd5-96zv4"
Oct 30 22:49:15.608349 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:15.608320    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e727c25-f834-4b00-aec6-9f3c7cb0b97b-dshm\") pod \"auth-test-model-5983ea-kserve-d4df66dd5-96zv4\" (UID: \"1e727c25-f834-4b00-aec6-9f3c7cb0b97b\") " pod="ai-tenant-e2e-shared-a-w0-5983ea/auth-test-model-5983ea-kserve-d4df66dd5-96zv4"
Oct 30 22:49:15.608774 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:15.608751    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e727c25-f834-4b00-aec6-9f3c7cb0b97b-tls-certs\") pod \"auth-test-model-5983ea-kserve-d4df66dd5-96zv4\" (UID: \"1e727c25-f834-4b00-aec6-9f3c7cb0b97b\") " pod="ai-tenant-e2e-shared-a-w0-5983ea/auth-test-model-5983ea-kserve-d4df66dd5-96zv4"
Oct 30 22:49:15.625792 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:15.625756    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbjmg\" (UniqueName: \"kubernetes.io/projected/1e727c25-f834-4b00-aec6-9f3c7cb0b97b-kube-api-access-hbjmg\") pod \"auth-test-model-5983ea-kserve-d4df66dd5-96zv4\" (UID: \"1e727c25-f834-4b00-aec6-9f3c7cb0b97b\") " pod="ai-tenant-e2e-shared-a-w0-5983ea/auth-test-model-5983ea-kserve-d4df66dd5-96zv4"
Oct 30 22:49:15.668140 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:15.668108    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-5983ea/auth-test-model-5983ea-kserve-d4df66dd5-96zv4"
Oct 30 22:49:15.982116 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:15.982083    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-5983ea/auth-test-model-5983ea-kserve-d4df66dd5-96zv4"]
Oct 30 22:49:15.985605 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:49:15.985560    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1e727c25_f834_4b00_aec6_9f3c7cb0b97b.slice/crio-2cee43baca258c605911f9130477fdb256209bdc28b5120e18700d098888660e WatchSource:0}: Error finding container 2cee43baca258c605911f9130477fdb256209bdc28b5120e18700d098888660e: Status 404 returned error can't find the container with id 2cee43baca258c605911f9130477fdb256209bdc28b5120e18700d098888660e
Oct 30 22:49:16.748123 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:16.748081    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-5983ea/auth-test-model-5983ea-kserve-d4df66dd5-96zv4" event={"ID":"1e727c25-f834-4b00-aec6-9f3c7cb0b97b","Type":"ContainerStarted","Data":"82f57cc8dcc27306103ee99ebc7f0cf34d7a1437428cf0405dc8eb30ac38c2f0"}
Oct 30 22:49:16.748580 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:16.748140    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-5983ea/auth-test-model-5983ea-kserve-d4df66dd5-96zv4" event={"ID":"1e727c25-f834-4b00-aec6-9f3c7cb0b97b","Type":"ContainerStarted","Data":"2cee43baca258c605911f9130477fdb256209bdc28b5120e18700d098888660e"}
Oct 30 22:49:16.789686 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:16.789625    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w0-5983ea/auth-test-model-5983ea-kserve-d4df66dd5-96zv4" podStartSLOduration=1.789607659 podStartE2EDuration="1.789607659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:49:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:49:16.785847961 +0000 UTC m=+2540.711774196" watchObservedRunningTime="2026-10-30 22:49:16.789607659 +0000 UTC m=+2540.715533867"
Oct 30 22:49:17.669126 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:17.669093    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-5983ea/auth-test-model-5983ea-kserve-d4df66dd5-96zv4"
Oct 30 22:49:23.166715 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:23.166681    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-bc48cc-6b58bfd8c7-xq5vn"]
Oct 30 22:49:23.167170 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:23.166914    4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bc48cc-6b58bfd8c7-xq5vn" podUID="08ebd2bf-5da3-418e-ac08-9584138d0b16" containerName="maas-api" containerID="cri-o://c29009e2f79e779de21ff2c57c6abe7f0ff0033811ca1bfd713f135f157b5691" gracePeriod=30
Oct 30 22:49:23.522586 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:23.522476    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bc48cc-6b58bfd8c7-xq5vn"
Oct 30 22:49:23.636914 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:23.636877    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08ebd2bf-5da3-418e-ac08-9584138d0b16-kube-api-access-nkbzr\" (UniqueName: \"kubernetes.io/projected/08ebd2bf-5da3-418e-ac08-9584138d0b16-kube-api-access-nkbzr\") pod \"08ebd2bf-5da3-418e-ac08-9584138d0b16\" (UID: \"08ebd2bf-5da3-418e-ac08-9584138d0b16\") "
Oct 30 22:49:23.638944 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:23.638908    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08ebd2bf-5da3-418e-ac08-9584138d0b16-metrics-certs\" (UniqueName: \"kubernetes.io/secret/08ebd2bf-5da3-418e-ac08-9584138d0b16-metrics-certs\") pod \"08ebd2bf-5da3-418e-ac08-9584138d0b16\" (UID: \"08ebd2bf-5da3-418e-ac08-9584138d0b16\") "
Oct 30 22:49:23.639257 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:23.639238    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08ebd2bf-5da3-418e-ac08-9584138d0b16-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08ebd2bf-5da3-418e-ac08-9584138d0b16-maas-api-tls\") pod \"08ebd2bf-5da3-418e-ac08-9584138d0b16\" (UID: \"08ebd2bf-5da3-418e-ac08-9584138d0b16\") "
Oct 30 22:49:23.643572 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:23.641185    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08ebd2bf-5da3-418e-ac08-9584138d0b16-openshift-service-ca" pod "08ebd2bf-5da3-418e-ac08-9584138d0b16" (UID: "08ebd2bf-5da3-418e-ac08-9584138d0b16"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:49:23.643572 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:23.639697    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08ebd2bf-5da3-418e-ac08-9584138d0b16-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08ebd2bf-5da3-418e-ac08-9584138d0b16-openshift-service-ca\") pod \"08ebd2bf-5da3-418e-ac08-9584138d0b16\" (UID: \"08ebd2bf-5da3-418e-ac08-9584138d0b16\") "
Oct 30 22:49:23.643572 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:23.641813    4477 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08ebd2bf-5da3-418e-ac08-9584138d0b16-openshift-service-ca\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:49:23.643572 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:23.643251    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08ebd2bf-5da3-418e-ac08-9584138d0b16-kube-api-access-nkbzr" pod "08ebd2bf-5da3-418e-ac08-9584138d0b16" (UID: "08ebd2bf-5da3-418e-ac08-9584138d0b16"). InnerVolumeSpecName "kube-api-access-nkbzr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:49:23.643572 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:23.643531    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08ebd2bf-5da3-418e-ac08-9584138d0b16-metrics-certs" pod "08ebd2bf-5da3-418e-ac08-9584138d0b16" (UID: "08ebd2bf-5da3-418e-ac08-9584138d0b16"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:49:23.643829 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:23.643705    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08ebd2bf-5da3-418e-ac08-9584138d0b16-maas-api-tls" pod "08ebd2bf-5da3-418e-ac08-9584138d0b16" (UID: "08ebd2bf-5da3-418e-ac08-9584138d0b16"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:49:23.743146 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:23.743063    4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkbzr\" (UniqueName: \"kubernetes.io/projected/08ebd2bf-5da3-418e-ac08-9584138d0b16-kube-api-access-nkbzr\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:49:23.743146 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:23.743090    4477 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08ebd2bf-5da3-418e-ac08-9584138d0b16-metrics-certs\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:49:23.743146 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:23.743119    4477 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08ebd2bf-5da3-418e-ac08-9584138d0b16-maas-api-tls\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:49:23.804255 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:23.804226    4477 generic.go:356] "Generic (PLEG): container finished" podID="08ebd2bf-5da3-418e-ac08-9584138d0b16" containerID="c29009e2f79e779de21ff2c57c6abe7f0ff0033811ca1bfd713f135f157b5691" exitCode=0
Oct 30 22:49:23.804412 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:23.804278    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bc48cc-6b58bfd8c7-xq5vn"
Oct 30 22:49:23.804412 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:23.804313    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bc48cc-6b58bfd8c7-xq5vn" event={"ID":"08ebd2bf-5da3-418e-ac08-9584138d0b16","Type":"ContainerDied","Data":"c29009e2f79e779de21ff2c57c6abe7f0ff0033811ca1bfd713f135f157b5691"}
Oct 30 22:49:23.804412 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:23.804386    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-bc48cc-6b58bfd8c7-xq5vn" event={"ID":"08ebd2bf-5da3-418e-ac08-9584138d0b16","Type":"ContainerDied","Data":"b0b374ebae01f3616e35fe031d5ed6b54c277e2fa32d45792fbdc6915c98f0f0"}
Oct 30 22:49:23.804557 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:23.804421    4477 scope.go:122] "RemoveContainer" containerID="c29009e2f79e779de21ff2c57c6abe7f0ff0033811ca1bfd713f135f157b5691"
Oct 30 22:49:23.815857 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:23.815841    4477 scope.go:122] "RemoveContainer" containerID="c29009e2f79e779de21ff2c57c6abe7f0ff0033811ca1bfd713f135f157b5691"
Oct 30 22:49:23.816131 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:49:23.816111    4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c29009e2f79e779de21ff2c57c6abe7f0ff0033811ca1bfd713f135f157b5691\": container with ID starting with c29009e2f79e779de21ff2c57c6abe7f0ff0033811ca1bfd713f135f157b5691 not found: ID does not exist" containerID="c29009e2f79e779de21ff2c57c6abe7f0ff0033811ca1bfd713f135f157b5691"
Oct 30 22:49:23.816199 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:23.816144    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c29009e2f79e779de21ff2c57c6abe7f0ff0033811ca1bfd713f135f157b5691"} err="failed to get container status \"c29009e2f79e779de21ff2c57c6abe7f0ff0033811ca1bfd713f135f157b5691\": rpc error: code = NotFound desc = could not find container \"c29009e2f79e779de21ff2c57c6abe7f0ff0033811ca1bfd713f135f157b5691\": container with ID starting with c29009e2f79e779de21ff2c57c6abe7f0ff0033811ca1bfd713f135f157b5691 not found: ID does not exist"
Oct 30 22:49:23.858464 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:23.858441    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-bc48cc-6b58bfd8c7-xq5vn"]
Oct 30 22:49:23.863296 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:23.863278    4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-bc48cc-6b58bfd8c7-xq5vn"]
Oct 30 22:49:24.193597 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:24.193564    4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08ebd2bf-5da3-418e-ac08-9584138d0b16" path="/var/lib/kubelet/pods/08ebd2bf-5da3-418e-ac08-9584138d0b16/volumes"
Oct 30 22:49:25.669151 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:25.669111    4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w0-5983ea/auth-test-model-5983ea-kserve-d4df66dd5-96zv4"
Oct 30 22:49:25.682111 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:25.682084    4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w0-5983ea/auth-test-model-5983ea-kserve-d4df66dd5-96zv4"
Oct 30 22:49:25.837616 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:25.835683    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-5983ea/auth-test-model-5983ea-kserve-d4df66dd5-96zv4"
Oct 30 22:49:26.266187 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:49:26.266151    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 22:49:26.916368 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:26.916332    4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-867f12/test-model-867f12-kserve-6694d95cf9-jcxgf" podUID="5f6746af-0eb9-4a46-a201-1e17626a9f81" containerName="main" containerID="cri-o://31211b50d977f546c6972dfed22b86eae487567be864c6d9c531d3ad35a50582" gracePeriod=45
Oct 30 22:49:27.195320 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:27.195297    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-867f12/test-model-867f12-kserve-6694d95cf9-jcxgf"
Oct 30 22:49:27.285521 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:27.285493    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f6746af-0eb9-4a46-a201-1e17626a9f81-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f6746af-0eb9-4a46-a201-1e17626a9f81-dshm\") pod \"5f6746af-0eb9-4a46-a201-1e17626a9f81\" (UID: \"5f6746af-0eb9-4a46-a201-1e17626a9f81\") "
Oct 30 22:49:27.285698 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:27.285626    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f6746af-0eb9-4a46-a201-1e17626a9f81-kube-api-access-w5bch\" (UniqueName: \"kubernetes.io/projected/5f6746af-0eb9-4a46-a201-1e17626a9f81-kube-api-access-w5bch\") pod \"5f6746af-0eb9-4a46-a201-1e17626a9f81\" (UID: \"5f6746af-0eb9-4a46-a201-1e17626a9f81\") "
Oct 30 22:49:27.285698 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:27.285688    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f6746af-0eb9-4a46-a201-1e17626a9f81-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f6746af-0eb9-4a46-a201-1e17626a9f81-model-cache\") pod \"5f6746af-0eb9-4a46-a201-1e17626a9f81\" (UID: \"5f6746af-0eb9-4a46-a201-1e17626a9f81\") "
Oct 30 22:49:27.285850 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:27.285720    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f6746af-0eb9-4a46-a201-1e17626a9f81-home\" (UniqueName: \"kubernetes.io/empty-dir/5f6746af-0eb9-4a46-a201-1e17626a9f81-home\") pod \"5f6746af-0eb9-4a46-a201-1e17626a9f81\" (UID: \"5f6746af-0eb9-4a46-a201-1e17626a9f81\") "
Oct 30 22:49:27.285919 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:27.285854    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f6746af-0eb9-4a46-a201-1e17626a9f81-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f6746af-0eb9-4a46-a201-1e17626a9f81-tmp-dir\") pod \"5f6746af-0eb9-4a46-a201-1e17626a9f81\" (UID: \"5f6746af-0eb9-4a46-a201-1e17626a9f81\") "
Oct 30 22:49:27.285919 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:27.285909    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f6746af-0eb9-4a46-a201-1e17626a9f81-tls-certs\" (UniqueName: \"kubernetes.io/secret/5f6746af-0eb9-4a46-a201-1e17626a9f81-tls-certs\") pod \"5f6746af-0eb9-4a46-a201-1e17626a9f81\" (UID: \"5f6746af-0eb9-4a46-a201-1e17626a9f81\") "
Oct 30 22:49:27.286064 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:27.285991    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f6746af-0eb9-4a46-a201-1e17626a9f81-model-cache" pod "5f6746af-0eb9-4a46-a201-1e17626a9f81" (UID: "5f6746af-0eb9-4a46-a201-1e17626a9f81"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:49:27.286145 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:27.286078    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f6746af-0eb9-4a46-a201-1e17626a9f81-home" pod "5f6746af-0eb9-4a46-a201-1e17626a9f81" (UID: "5f6746af-0eb9-4a46-a201-1e17626a9f81"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:49:27.286201 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:27.286165    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f6746af-0eb9-4a46-a201-1e17626a9f81-tmp-dir" pod "5f6746af-0eb9-4a46-a201-1e17626a9f81" (UID: "5f6746af-0eb9-4a46-a201-1e17626a9f81"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:49:27.286538 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:27.286514    4477 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f6746af-0eb9-4a46-a201-1e17626a9f81-model-cache\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:49:27.286538 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:27.286539    4477 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f6746af-0eb9-4a46-a201-1e17626a9f81-home\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:49:27.286703 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:27.286554    4477 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f6746af-0eb9-4a46-a201-1e17626a9f81-tmp-dir\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:49:27.287846 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:27.287823    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f6746af-0eb9-4a46-a201-1e17626a9f81-dshm" pod "5f6746af-0eb9-4a46-a201-1e17626a9f81" (UID: "5f6746af-0eb9-4a46-a201-1e17626a9f81"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:49:27.287932 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:27.287916    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f6746af-0eb9-4a46-a201-1e17626a9f81-tls-certs" pod "5f6746af-0eb9-4a46-a201-1e17626a9f81" (UID: "5f6746af-0eb9-4a46-a201-1e17626a9f81"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:49:27.287999 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:27.287978    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f6746af-0eb9-4a46-a201-1e17626a9f81-kube-api-access-w5bch" pod "5f6746af-0eb9-4a46-a201-1e17626a9f81" (UID: "5f6746af-0eb9-4a46-a201-1e17626a9f81"). InnerVolumeSpecName "kube-api-access-w5bch". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:49:27.387961 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:27.387938    4477 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f6746af-0eb9-4a46-a201-1e17626a9f81-dshm\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:49:27.387961 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:27.387962    4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5bch\" (UniqueName: \"kubernetes.io/projected/5f6746af-0eb9-4a46-a201-1e17626a9f81-kube-api-access-w5bch\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:49:27.388098 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:27.387970    4477 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f6746af-0eb9-4a46-a201-1e17626a9f81-tls-certs\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:49:27.830566 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:27.830535    4477 generic.go:356] "Generic (PLEG): container finished" podID="5f6746af-0eb9-4a46-a201-1e17626a9f81" containerID="31211b50d977f546c6972dfed22b86eae487567be864c6d9c531d3ad35a50582" exitCode=0
Oct 30 22:49:27.830739 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:27.830576    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-867f12/test-model-867f12-kserve-6694d95cf9-jcxgf"
Oct 30 22:49:27.830739 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:27.830619    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-867f12/test-model-867f12-kserve-6694d95cf9-jcxgf" event={"ID":"5f6746af-0eb9-4a46-a201-1e17626a9f81","Type":"ContainerDied","Data":"31211b50d977f546c6972dfed22b86eae487567be864c6d9c531d3ad35a50582"}
Oct 30 22:49:27.830739 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:27.830659    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-867f12/test-model-867f12-kserve-6694d95cf9-jcxgf" event={"ID":"5f6746af-0eb9-4a46-a201-1e17626a9f81","Type":"ContainerDied","Data":"e5a72d53a33090efd1e94dbf67f02a664dbc5fcbb6f704c42328f3f86f029f03"}
Oct 30 22:49:27.830739 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:27.830676    4477 scope.go:122] "RemoveContainer" containerID="31211b50d977f546c6972dfed22b86eae487567be864c6d9c531d3ad35a50582"
Oct 30 22:49:27.840989 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:27.840973    4477 scope.go:122] "RemoveContainer" containerID="31211b50d977f546c6972dfed22b86eae487567be864c6d9c531d3ad35a50582"
Oct 30 22:49:27.841284 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:49:27.841263    4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31211b50d977f546c6972dfed22b86eae487567be864c6d9c531d3ad35a50582\": container with ID starting with 31211b50d977f546c6972dfed22b86eae487567be864c6d9c531d3ad35a50582 not found: ID does not exist" containerID="31211b50d977f546c6972dfed22b86eae487567be864c6d9c531d3ad35a50582"
Oct 30 22:49:27.841328 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:27.841291    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31211b50d977f546c6972dfed22b86eae487567be864c6d9c531d3ad35a50582"} err="failed to get container status \"31211b50d977f546c6972dfed22b86eae487567be864c6d9c531d3ad35a50582\": rpc error: code = NotFound desc = could not find container \"31211b50d977f546c6972dfed22b86eae487567be864c6d9c531d3ad35a50582\": container with ID starting with 31211b50d977f546c6972dfed22b86eae487567be864c6d9c531d3ad35a50582 not found: ID does not exist"
Oct 30 22:49:27.881286 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:27.881259    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-867f12/test-model-867f12-kserve-6694d95cf9-jcxgf"]
Oct 30 22:49:27.893009 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:27.892986    4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-867f12/test-model-867f12-kserve-6694d95cf9-jcxgf"]
Oct 30 22:49:28.155554 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:28.155518    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-867f12/test-model-867f12-kserve-6694d95cf9-jcxgf" podUID="5f6746af-0eb9-4a46-a201-1e17626a9f81" containerName="main" probeResult="failure" output="Get \"https://10.134.0.128:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:49:28.193604 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:28.193574    4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f6746af-0eb9-4a46-a201-1e17626a9f81" path="/var/lib/kubelet/pods/5f6746af-0eb9-4a46-a201-1e17626a9f81/volumes"
Oct 30 22:49:45.230077 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:45.229979    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-3acfdb-76fffb9796-mqnhp"]
Oct 30 22:49:45.230673 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:45.230291    4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3acfdb-76fffb9796-mqnhp" podUID="0ede416f-63b4-4f4b-802f-22f1f3183e9f" containerName="maas-api" containerID="cri-o://8f17694c7b720a70b3bf5138aadb0e35d8fbfe90109c7873dbb8f1e36fa2ff8e" gracePeriod=30
Oct 30 22:49:45.658874 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:45.658565    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3acfdb-76fffb9796-mqnhp"
Oct 30 22:49:45.726577 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:45.726546    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ede416f-63b4-4f4b-802f-22f1f3183e9f-kube-api-access-x4zbj\" (UniqueName: \"kubernetes.io/projected/0ede416f-63b4-4f4b-802f-22f1f3183e9f-kube-api-access-x4zbj\") pod \"0ede416f-63b4-4f4b-802f-22f1f3183e9f\" (UID: \"0ede416f-63b4-4f4b-802f-22f1f3183e9f\") "
Oct 30 22:49:45.726882 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:45.726862    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ede416f-63b4-4f4b-802f-22f1f3183e9f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ede416f-63b4-4f4b-802f-22f1f3183e9f-openshift-service-ca\") pod \"0ede416f-63b4-4f4b-802f-22f1f3183e9f\" (UID: \"0ede416f-63b4-4f4b-802f-22f1f3183e9f\") "
Oct 30 22:49:45.727108 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:45.727091    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ede416f-63b4-4f4b-802f-22f1f3183e9f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ede416f-63b4-4f4b-802f-22f1f3183e9f-maas-api-tls\") pod \"0ede416f-63b4-4f4b-802f-22f1f3183e9f\" (UID: \"0ede416f-63b4-4f4b-802f-22f1f3183e9f\") "
Oct 30 22:49:45.727648 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:45.727515    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ede416f-63b4-4f4b-802f-22f1f3183e9f-openshift-service-ca" pod "0ede416f-63b4-4f4b-802f-22f1f3183e9f" (UID: "0ede416f-63b4-4f4b-802f-22f1f3183e9f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:49:45.727752 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:45.727623    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ede416f-63b4-4f4b-802f-22f1f3183e9f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ede416f-63b4-4f4b-802f-22f1f3183e9f-metrics-certs\") pod \"0ede416f-63b4-4f4b-802f-22f1f3183e9f\" (UID: \"0ede416f-63b4-4f4b-802f-22f1f3183e9f\") "
Oct 30 22:49:45.728316 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:45.728292    4477 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ede416f-63b4-4f4b-802f-22f1f3183e9f-openshift-service-ca\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:49:45.735071 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:45.729821    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ede416f-63b4-4f4b-802f-22f1f3183e9f-maas-api-tls" pod "0ede416f-63b4-4f4b-802f-22f1f3183e9f" (UID: "0ede416f-63b4-4f4b-802f-22f1f3183e9f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:49:45.735071 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:45.730079    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ede416f-63b4-4f4b-802f-22f1f3183e9f-kube-api-access-x4zbj" pod "0ede416f-63b4-4f4b-802f-22f1f3183e9f" (UID: "0ede416f-63b4-4f4b-802f-22f1f3183e9f"). InnerVolumeSpecName "kube-api-access-x4zbj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:49:45.735335 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:45.735320    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ede416f-63b4-4f4b-802f-22f1f3183e9f-metrics-certs" pod "0ede416f-63b4-4f4b-802f-22f1f3183e9f" (UID: "0ede416f-63b4-4f4b-802f-22f1f3183e9f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:49:45.829604 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:45.829527    4477 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ede416f-63b4-4f4b-802f-22f1f3183e9f-maas-api-tls\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:49:45.829604 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:45.829561    4477 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ede416f-63b4-4f4b-802f-22f1f3183e9f-metrics-certs\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:49:45.829604 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:45.829578    4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4zbj\" (UniqueName: \"kubernetes.io/projected/0ede416f-63b4-4f4b-802f-22f1f3183e9f-kube-api-access-x4zbj\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:49:45.932966 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:45.932938    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-b03b33/auth-test-model-b03b33-kserve-d99bb5cb9-fzz5k"]
Oct 30 22:49:45.933500 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:45.933487    4477 cpu_manager.go:414] "containerMap: removing container" podUID="0ede416f-63b4-4f4b-802f-22f1f3183e9f" containerName="maas-api"
Oct 30 22:49:45.933550 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:45.933502    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ede416f-63b4-4f4b-802f-22f1f3183e9f" containerName="maas-api"
Oct 30 22:49:45.933550 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:45.933512    4477 cpu_manager.go:414] "containerMap: removing container" podUID="08ebd2bf-5da3-418e-ac08-9584138d0b16" containerName="maas-api"
Oct 30 22:49:45.933550 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:45.933518    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08ebd2bf-5da3-418e-ac08-9584138d0b16" containerName="maas-api"
Oct 30 22:49:45.933550 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:45.933539    4477 cpu_manager.go:414] "containerMap: removing container" podUID="5f6746af-0eb9-4a46-a201-1e17626a9f81" containerName="main"
Oct 30 22:49:45.933550 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:45.933544    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f6746af-0eb9-4a46-a201-1e17626a9f81" containerName="main"
Oct 30 22:49:45.933705 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:45.933626    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ede416f-63b4-4f4b-802f-22f1f3183e9f" containerName="maas-api"
Oct 30 22:49:45.933705 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:45.933633    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="08ebd2bf-5da3-418e-ac08-9584138d0b16" containerName="maas-api"
Oct 30 22:49:45.933705 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:45.933655    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f6746af-0eb9-4a46-a201-1e17626a9f81" containerName="main"
Oct 30 22:49:45.937309 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:45.937287    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-b03b33/auth-test-model-b03b33-kserve-d99bb5cb9-fzz5k"
Oct 30 22:49:45.947364 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:45.947110    4477 generic.go:356] "Generic (PLEG): container finished" podID="0ede416f-63b4-4f4b-802f-22f1f3183e9f" containerID="8f17694c7b720a70b3bf5138aadb0e35d8fbfe90109c7873dbb8f1e36fa2ff8e" exitCode=0
Oct 30 22:49:45.947364 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:45.947225    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3acfdb-76fffb9796-mqnhp" event={"ID":"0ede416f-63b4-4f4b-802f-22f1f3183e9f","Type":"ContainerDied","Data":"8f17694c7b720a70b3bf5138aadb0e35d8fbfe90109c7873dbb8f1e36fa2ff8e"}
Oct 30 22:49:45.947364 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:45.947271    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3acfdb-76fffb9796-mqnhp" event={"ID":"0ede416f-63b4-4f4b-802f-22f1f3183e9f","Type":"ContainerDied","Data":"79f2c3d7b301a896b9b12af3a632328aaebfade6e602db5a5ba4546f700b85e5"}
Oct 30 22:49:45.947364 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:45.947301    4477 scope.go:122] "RemoveContainer" containerID="8f17694c7b720a70b3bf5138aadb0e35d8fbfe90109c7873dbb8f1e36fa2ff8e"
Oct 30 22:49:45.947631 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:45.947462    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3acfdb-76fffb9796-mqnhp"
Oct 30 22:49:45.959340 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:45.959315    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-b03b33/auth-test-model-b03b33-kserve-d99bb5cb9-fzz5k"]
Oct 30 22:49:45.959484 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:45.959432    4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-b03b33\"/\"openshift-service-ca.crt\""
Oct 30 22:49:45.959557 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:45.959506    4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-b03b33\"/\"kube-root-ca.crt\""
Oct 30 22:49:45.963083 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:45.961413    4477 scope.go:122] "RemoveContainer" containerID="8f17694c7b720a70b3bf5138aadb0e35d8fbfe90109c7873dbb8f1e36fa2ff8e"
Oct 30 22:49:45.963083 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:49:45.961810    4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f17694c7b720a70b3bf5138aadb0e35d8fbfe90109c7873dbb8f1e36fa2ff8e\": container with ID starting with 8f17694c7b720a70b3bf5138aadb0e35d8fbfe90109c7873dbb8f1e36fa2ff8e not found: ID does not exist" containerID="8f17694c7b720a70b3bf5138aadb0e35d8fbfe90109c7873dbb8f1e36fa2ff8e"
Oct 30 22:49:45.963083 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:45.961843    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f17694c7b720a70b3bf5138aadb0e35d8fbfe90109c7873dbb8f1e36fa2ff8e"} err="failed to get container status \"8f17694c7b720a70b3bf5138aadb0e35d8fbfe90109c7873dbb8f1e36fa2ff8e\": rpc error: code = NotFound desc = could not find container \"8f17694c7b720a70b3bf5138aadb0e35d8fbfe90109c7873dbb8f1e36fa2ff8e\": container with ID starting with 8f17694c7b720a70b3bf5138aadb0e35d8fbfe90109c7873dbb8f1e36fa2ff8e not found: ID does not exist"
Oct 30 22:49:46.003610 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:46.003577    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-b03b33\"/\"default-dockercfg-r9btf\""
Oct 30 22:49:46.003610 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:46.003587    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-b03b33\"/\"auth-test-model-b03b33-kserve-self-signed-certs\""
Oct 30 22:49:46.022406 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:46.022381    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-3acfdb-76fffb9796-mqnhp"]
Oct 30 22:49:46.029825 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:46.029797    4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-3acfdb-76fffb9796-mqnhp"]
Oct 30 22:49:46.032314 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:46.032289    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae4a262e-38d8-493c-95f8-1bf666eff305-tmp-dir\") pod \"auth-test-model-b03b33-kserve-d99bb5cb9-fzz5k\" (UID: \"ae4a262e-38d8-493c-95f8-1bf666eff305\") " pod="ai-tenant-e2e-shared-b-w4-b03b33/auth-test-model-b03b33-kserve-d99bb5cb9-fzz5k"
Oct 30 22:49:46.032434 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:46.032346    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l85f6\" (UniqueName: \"kubernetes.io/projected/ae4a262e-38d8-493c-95f8-1bf666eff305-kube-api-access-l85f6\") pod \"auth-test-model-b03b33-kserve-d99bb5cb9-fzz5k\" (UID: \"ae4a262e-38d8-493c-95f8-1bf666eff305\") " pod="ai-tenant-e2e-shared-b-w4-b03b33/auth-test-model-b03b33-kserve-d99bb5cb9-fzz5k"
Oct 30 22:49:46.032434 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:46.032416    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae4a262e-38d8-493c-95f8-1bf666eff305-home\") pod \"auth-test-model-b03b33-kserve-d99bb5cb9-fzz5k\" (UID: \"ae4a262e-38d8-493c-95f8-1bf666eff305\") " pod="ai-tenant-e2e-shared-b-w4-b03b33/auth-test-model-b03b33-kserve-d99bb5cb9-fzz5k"
Oct 30 22:49:46.032533 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:46.032461    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae4a262e-38d8-493c-95f8-1bf666eff305-dshm\") pod \"auth-test-model-b03b33-kserve-d99bb5cb9-fzz5k\" (UID: \"ae4a262e-38d8-493c-95f8-1bf666eff305\") " pod="ai-tenant-e2e-shared-b-w4-b03b33/auth-test-model-b03b33-kserve-d99bb5cb9-fzz5k"
Oct 30 22:49:46.032585 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:46.032574    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae4a262e-38d8-493c-95f8-1bf666eff305-model-cache\") pod \"auth-test-model-b03b33-kserve-d99bb5cb9-fzz5k\" (UID: \"ae4a262e-38d8-493c-95f8-1bf666eff305\") " pod="ai-tenant-e2e-shared-b-w4-b03b33/auth-test-model-b03b33-kserve-d99bb5cb9-fzz5k"
Oct 30 22:49:46.032632 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:46.032602    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae4a262e-38d8-493c-95f8-1bf666eff305-tls-certs\") pod \"auth-test-model-b03b33-kserve-d99bb5cb9-fzz5k\" (UID: \"ae4a262e-38d8-493c-95f8-1bf666eff305\") " pod="ai-tenant-e2e-shared-b-w4-b03b33/auth-test-model-b03b33-kserve-d99bb5cb9-fzz5k"
Oct 30 22:49:46.134360 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:46.134326    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae4a262e-38d8-493c-95f8-1bf666eff305-tmp-dir\") pod \"auth-test-model-b03b33-kserve-d99bb5cb9-fzz5k\" (UID: \"ae4a262e-38d8-493c-95f8-1bf666eff305\") " pod="ai-tenant-e2e-shared-b-w4-b03b33/auth-test-model-b03b33-kserve-d99bb5cb9-fzz5k"
Oct 30 22:49:46.134609 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:46.134587    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l85f6\" (UniqueName: \"kubernetes.io/projected/ae4a262e-38d8-493c-95f8-1bf666eff305-kube-api-access-l85f6\") pod \"auth-test-model-b03b33-kserve-d99bb5cb9-fzz5k\" (UID: \"ae4a262e-38d8-493c-95f8-1bf666eff305\") " pod="ai-tenant-e2e-shared-b-w4-b03b33/auth-test-model-b03b33-kserve-d99bb5cb9-fzz5k"
Oct 30 22:49:46.134719 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:46.134635    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae4a262e-38d8-493c-95f8-1bf666eff305-home\") pod \"auth-test-model-b03b33-kserve-d99bb5cb9-fzz5k\" (UID: \"ae4a262e-38d8-493c-95f8-1bf666eff305\") " pod="ai-tenant-e2e-shared-b-w4-b03b33/auth-test-model-b03b33-kserve-d99bb5cb9-fzz5k"
Oct 30 22:49:46.134719 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:46.134699    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae4a262e-38d8-493c-95f8-1bf666eff305-dshm\") pod \"auth-test-model-b03b33-kserve-d99bb5cb9-fzz5k\" (UID: \"ae4a262e-38d8-493c-95f8-1bf666eff305\") " pod="ai-tenant-e2e-shared-b-w4-b03b33/auth-test-model-b03b33-kserve-d99bb5cb9-fzz5k"
Oct 30 22:49:46.134833 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:46.134780    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae4a262e-38d8-493c-95f8-1bf666eff305-model-cache\") pod \"auth-test-model-b03b33-kserve-d99bb5cb9-fzz5k\" (UID: \"ae4a262e-38d8-493c-95f8-1bf666eff305\") " pod="ai-tenant-e2e-shared-b-w4-b03b33/auth-test-model-b03b33-kserve-d99bb5cb9-fzz5k"
Oct 30 22:49:46.134833 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:46.134782    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae4a262e-38d8-493c-95f8-1bf666eff305-tmp-dir\") pod \"auth-test-model-b03b33-kserve-d99bb5cb9-fzz5k\" (UID: \"ae4a262e-38d8-493c-95f8-1bf666eff305\") " pod="ai-tenant-e2e-shared-b-w4-b03b33/auth-test-model-b03b33-kserve-d99bb5cb9-fzz5k"
Oct 30 22:49:46.134833 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:46.134824    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae4a262e-38d8-493c-95f8-1bf666eff305-tls-certs\") pod \"auth-test-model-b03b33-kserve-d99bb5cb9-fzz5k\" (UID: \"ae4a262e-38d8-493c-95f8-1bf666eff305\") " pod="ai-tenant-e2e-shared-b-w4-b03b33/auth-test-model-b03b33-kserve-d99bb5cb9-fzz5k"
Oct 30 22:49:46.135325 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:46.135292    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae4a262e-38d8-493c-95f8-1bf666eff305-home\") pod \"auth-test-model-b03b33-kserve-d99bb5cb9-fzz5k\" (UID: \"ae4a262e-38d8-493c-95f8-1bf666eff305\") " pod="ai-tenant-e2e-shared-b-w4-b03b33/auth-test-model-b03b33-kserve-d99bb5cb9-fzz5k"
Oct 30 22:49:46.135447 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:46.135340    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae4a262e-38d8-493c-95f8-1bf666eff305-model-cache\") pod \"auth-test-model-b03b33-kserve-d99bb5cb9-fzz5k\" (UID: \"ae4a262e-38d8-493c-95f8-1bf666eff305\") " pod="ai-tenant-e2e-shared-b-w4-b03b33/auth-test-model-b03b33-kserve-d99bb5cb9-fzz5k"
Oct 30 22:49:46.137218 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:46.137190    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae4a262e-38d8-493c-95f8-1bf666eff305-dshm\") pod \"auth-test-model-b03b33-kserve-d99bb5cb9-fzz5k\" (UID: \"ae4a262e-38d8-493c-95f8-1bf666eff305\") " pod="ai-tenant-e2e-shared-b-w4-b03b33/auth-test-model-b03b33-kserve-d99bb5cb9-fzz5k"
Oct 30 22:49:46.137400 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:46.137381    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae4a262e-38d8-493c-95f8-1bf666eff305-tls-certs\") pod \"auth-test-model-b03b33-kserve-d99bb5cb9-fzz5k\" (UID: \"ae4a262e-38d8-493c-95f8-1bf666eff305\") " pod="ai-tenant-e2e-shared-b-w4-b03b33/auth-test-model-b03b33-kserve-d99bb5cb9-fzz5k"
Oct 30 22:49:46.158310 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:46.158274    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l85f6\" (UniqueName: \"kubernetes.io/projected/ae4a262e-38d8-493c-95f8-1bf666eff305-kube-api-access-l85f6\") pod \"auth-test-model-b03b33-kserve-d99bb5cb9-fzz5k\" (UID: \"ae4a262e-38d8-493c-95f8-1bf666eff305\") " pod="ai-tenant-e2e-shared-b-w4-b03b33/auth-test-model-b03b33-kserve-d99bb5cb9-fzz5k"
Oct 30 22:49:46.198761 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:46.198685    4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ede416f-63b4-4f4b-802f-22f1f3183e9f" path="/var/lib/kubelet/pods/0ede416f-63b4-4f4b-802f-22f1f3183e9f/volumes"
Oct 30 22:49:46.254660 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:46.254617    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-b03b33/auth-test-model-b03b33-kserve-d99bb5cb9-fzz5k"
Oct 30 22:49:46.605137 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:46.604818    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-b03b33/auth-test-model-b03b33-kserve-d99bb5cb9-fzz5k"]
Oct 30 22:49:46.958116 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:46.958083    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-b03b33/auth-test-model-b03b33-kserve-d99bb5cb9-fzz5k" event={"ID":"ae4a262e-38d8-493c-95f8-1bf666eff305","Type":"ContainerStarted","Data":"cb6e55d9badb01455dbb30f16c9da20ea07c3db38284d38cff68f4feeffae420"}
Oct 30 22:49:46.958116 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:46.958121    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-b03b33/auth-test-model-b03b33-kserve-d99bb5cb9-fzz5k" event={"ID":"ae4a262e-38d8-493c-95f8-1bf666eff305","Type":"ContainerStarted","Data":"7f6a4df5b9f77932099b3b166b3657c2b38d727b3ebf524507a9042cf71e1f02"}
Oct 30 22:49:47.000387 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:47.000327    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-b03b33/auth-test-model-b03b33-kserve-d99bb5cb9-fzz5k" podStartSLOduration=2.000314379 podStartE2EDuration="2.000314379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:49:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:49:46.997211306 +0000 UTC m=+2570.923137514" watchObservedRunningTime="2026-10-30 22:49:47.000314379 +0000 UTC m=+2570.926240585"
Oct 30 22:49:47.255138 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:47.255050    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-b03b33/auth-test-model-b03b33-kserve-d99bb5cb9-fzz5k"
Oct 30 22:49:56.254786 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:56.254753    4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-b03b33/auth-test-model-b03b33-kserve-d99bb5cb9-fzz5k"
Oct 30 22:49:56.268904 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:56.268878    4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-b03b33/auth-test-model-b03b33-kserve-d99bb5cb9-fzz5k"
Oct 30 22:49:56.270668 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:49:56.270641    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 22:49:57.035479 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:49:57.035452    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-b03b33/auth-test-model-b03b33-kserve-d99bb5cb9-fzz5k"
Oct 30 22:50:03.887675 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:03.887637    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-867f12-8467f5bbc9-xfx5x"]
Oct 30 22:50:03.888171 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:03.887898    4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-867f12-8467f5bbc9-xfx5x" podUID="5ff74f16-8919-4c9c-9b49-f333618cdf65" containerName="maas-api" containerID="cri-o://2c10f038af7e963e324f19a53f8467b3f17a0f4fd09011d92c95d07f25babe5f" gracePeriod=30
Oct 30 22:50:04.078123 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:04.078087    4477 generic.go:356] "Generic (PLEG): container finished" podID="5ff74f16-8919-4c9c-9b49-f333618cdf65" containerID="2c10f038af7e963e324f19a53f8467b3f17a0f4fd09011d92c95d07f25babe5f" exitCode=0
Oct 30 22:50:04.078431 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:04.078406    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-867f12-8467f5bbc9-xfx5x" event={"ID":"5ff74f16-8919-4c9c-9b49-f333618cdf65","Type":"ContainerDied","Data":"2c10f038af7e963e324f19a53f8467b3f17a0f4fd09011d92c95d07f25babe5f"}
Oct 30 22:50:04.204674 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:04.204642    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-867f12-8467f5bbc9-xfx5x"
Oct 30 22:50:04.274498 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:04.274456    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ff74f16-8919-4c9c-9b49-f333618cdf65-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ff74f16-8919-4c9c-9b49-f333618cdf65-maas-api-tls\") pod \"5ff74f16-8919-4c9c-9b49-f333618cdf65\" (UID: \"5ff74f16-8919-4c9c-9b49-f333618cdf65\") "
Oct 30 22:50:04.274684 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:04.274552    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ff74f16-8919-4c9c-9b49-f333618cdf65-kube-api-access-4cnn4\" (UniqueName: \"kubernetes.io/projected/5ff74f16-8919-4c9c-9b49-f333618cdf65-kube-api-access-4cnn4\") pod \"5ff74f16-8919-4c9c-9b49-f333618cdf65\" (UID: \"5ff74f16-8919-4c9c-9b49-f333618cdf65\") "
Oct 30 22:50:04.274684 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:04.274650    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ff74f16-8919-4c9c-9b49-f333618cdf65-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ff74f16-8919-4c9c-9b49-f333618cdf65-metrics-certs\") pod \"5ff74f16-8919-4c9c-9b49-f333618cdf65\" (UID: \"5ff74f16-8919-4c9c-9b49-f333618cdf65\") "
Oct 30 22:50:04.274812 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:04.274690    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ff74f16-8919-4c9c-9b49-f333618cdf65-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ff74f16-8919-4c9c-9b49-f333618cdf65-openshift-service-ca\") pod \"5ff74f16-8919-4c9c-9b49-f333618cdf65\" (UID: \"5ff74f16-8919-4c9c-9b49-f333618cdf65\") "
Oct 30 22:50:04.275470 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:04.275443    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ff74f16-8919-4c9c-9b49-f333618cdf65-openshift-service-ca" pod "5ff74f16-8919-4c9c-9b49-f333618cdf65" (UID: "5ff74f16-8919-4c9c-9b49-f333618cdf65"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:50:04.278032 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:04.277992    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ff74f16-8919-4c9c-9b49-f333618cdf65-metrics-certs" pod "5ff74f16-8919-4c9c-9b49-f333618cdf65" (UID: "5ff74f16-8919-4c9c-9b49-f333618cdf65"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:50:04.280701 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:04.280658    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ff74f16-8919-4c9c-9b49-f333618cdf65-maas-api-tls" pod "5ff74f16-8919-4c9c-9b49-f333618cdf65" (UID: "5ff74f16-8919-4c9c-9b49-f333618cdf65"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:50:04.283216 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:04.283182    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ff74f16-8919-4c9c-9b49-f333618cdf65-kube-api-access-4cnn4" pod "5ff74f16-8919-4c9c-9b49-f333618cdf65" (UID: "5ff74f16-8919-4c9c-9b49-f333618cdf65"). InnerVolumeSpecName "kube-api-access-4cnn4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:50:04.375981 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:04.375940    4477 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ff74f16-8919-4c9c-9b49-f333618cdf65-maas-api-tls\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:50:04.375981 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:04.375975    4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cnn4\" (UniqueName: \"kubernetes.io/projected/5ff74f16-8919-4c9c-9b49-f333618cdf65-kube-api-access-4cnn4\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:50:04.375981 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:04.375988    4477 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ff74f16-8919-4c9c-9b49-f333618cdf65-metrics-certs\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:50:04.376283 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:04.376000    4477 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ff74f16-8919-4c9c-9b49-f333618cdf65-openshift-service-ca\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:50:05.084995 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:05.084968    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-867f12-8467f5bbc9-xfx5x"
Oct 30 22:50:05.084995 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:05.084980    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-867f12-8467f5bbc9-xfx5x" event={"ID":"5ff74f16-8919-4c9c-9b49-f333618cdf65","Type":"ContainerDied","Data":"04208f4db4541fb84c808f4cf1037a9cdb3a262c99381615ee7b05688e1a71a6"}
Oct 30 22:50:05.085481 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:05.085059    4477 scope.go:122] "RemoveContainer" containerID="2c10f038af7e963e324f19a53f8467b3f17a0f4fd09011d92c95d07f25babe5f"
Oct 30 22:50:05.145906 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:05.145876    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-867f12-8467f5bbc9-xfx5x"]
Oct 30 22:50:05.151941 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:05.151915    4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-867f12-8467f5bbc9-xfx5x"]
Oct 30 22:50:06.193167 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:06.193133    4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ff74f16-8919-4c9c-9b49-f333618cdf65" path="/var/lib/kubelet/pods/5ff74f16-8919-4c9c-9b49-f333618cdf65/volumes"
Oct 30 22:50:11.983195 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:11.983155    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-znd9z"]
Oct 30 22:50:11.983690 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:11.983359    4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-znd9z" podUID="2281431c-56b4-475a-a7dc-35c1bbe04e1a" containerName="istio-proxy" containerID="cri-o://d0afc0d62e8f97d2464f010d461c8190334728789c0ece70c2b3a208ee4fbbfb" gracePeriod=30
Oct 30 22:50:17.168501 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:17.168114    4477 generic.go:356] "Generic (PLEG): container finished" podID="2281431c-56b4-475a-a7dc-35c1bbe04e1a" containerID="d0afc0d62e8f97d2464f010d461c8190334728789c0ece70c2b3a208ee4fbbfb" exitCode=0
Oct 30 22:50:17.168501 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:17.168173    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-znd9z" event={"ID":"2281431c-56b4-475a-a7dc-35c1bbe04e1a","Type":"ContainerDied","Data":"d0afc0d62e8f97d2464f010d461c8190334728789c0ece70c2b3a208ee4fbbfb"}
Oct 30 22:50:17.255003 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:17.254941    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-znd9z"
Oct 30 22:50:17.430084 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:17.430028    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2281431c-56b4-475a-a7dc-35c1bbe04e1a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2281431c-56b4-475a-a7dc-35c1bbe04e1a-istio-podinfo\") pod \"2281431c-56b4-475a-a7dc-35c1bbe04e1a\" (UID: \"2281431c-56b4-475a-a7dc-35c1bbe04e1a\") "
Oct 30 22:50:17.430271 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:17.430111    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2281431c-56b4-475a-a7dc-35c1bbe04e1a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2281431c-56b4-475a-a7dc-35c1bbe04e1a-istio-data\") pod \"2281431c-56b4-475a-a7dc-35c1bbe04e1a\" (UID: \"2281431c-56b4-475a-a7dc-35c1bbe04e1a\") "
Oct 30 22:50:17.430271 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:17.430142    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2281431c-56b4-475a-a7dc-35c1bbe04e1a-kube-api-access-dwqnl\" (UniqueName: \"kubernetes.io/projected/2281431c-56b4-475a-a7dc-35c1bbe04e1a-kube-api-access-dwqnl\") pod \"2281431c-56b4-475a-a7dc-35c1bbe04e1a\" (UID: \"2281431c-56b4-475a-a7dc-35c1bbe04e1a\") "
Oct 30 22:50:17.430271 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:17.430233    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2281431c-56b4-475a-a7dc-35c1bbe04e1a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2281431c-56b4-475a-a7dc-35c1bbe04e1a-workload-socket\") pod \"2281431c-56b4-475a-a7dc-35c1bbe04e1a\" (UID: \"2281431c-56b4-475a-a7dc-35c1bbe04e1a\") "
Oct 30 22:50:17.430271 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:17.430261    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2281431c-56b4-475a-a7dc-35c1bbe04e1a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2281431c-56b4-475a-a7dc-35c1bbe04e1a-istio-envoy\") pod \"2281431c-56b4-475a-a7dc-35c1bbe04e1a\" (UID: \"2281431c-56b4-475a-a7dc-35c1bbe04e1a\") "
Oct 30 22:50:17.430496 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:17.430378    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2281431c-56b4-475a-a7dc-35c1bbe04e1a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2281431c-56b4-475a-a7dc-35c1bbe04e1a-istiod-ca-cert\") pod \"2281431c-56b4-475a-a7dc-35c1bbe04e1a\" (UID: \"2281431c-56b4-475a-a7dc-35c1bbe04e1a\") "
Oct 30 22:50:17.430496 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:17.430433    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2281431c-56b4-475a-a7dc-35c1bbe04e1a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2281431c-56b4-475a-a7dc-35c1bbe04e1a-credential-socket\") pod \"2281431c-56b4-475a-a7dc-35c1bbe04e1a\" (UID: \"2281431c-56b4-475a-a7dc-35c1bbe04e1a\") "
Oct 30 22:50:17.430496 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:17.430457    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2281431c-56b4-475a-a7dc-35c1bbe04e1a-istio-data" pod "2281431c-56b4-475a-a7dc-35c1bbe04e1a" (UID: "2281431c-56b4-475a-a7dc-35c1bbe04e1a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:50:17.430496 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:17.430471    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2281431c-56b4-475a-a7dc-35c1bbe04e1a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2281431c-56b4-475a-a7dc-35c1bbe04e1a-workload-certs\") pod \"2281431c-56b4-475a-a7dc-35c1bbe04e1a\" (UID: \"2281431c-56b4-475a-a7dc-35c1bbe04e1a\") "
Oct 30 22:50:17.430717 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:17.430507    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2281431c-56b4-475a-a7dc-35c1bbe04e1a-workload-socket" pod "2281431c-56b4-475a-a7dc-35c1bbe04e1a" (UID: "2281431c-56b4-475a-a7dc-35c1bbe04e1a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:50:17.430717 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:17.430548    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2281431c-56b4-475a-a7dc-35c1bbe04e1a-istio-token\" (UniqueName: \"kubernetes.io/projected/2281431c-56b4-475a-a7dc-35c1bbe04e1a-istio-token\") pod \"2281431c-56b4-475a-a7dc-35c1bbe04e1a\" (UID: \"2281431c-56b4-475a-a7dc-35c1bbe04e1a\") "
Oct 30 22:50:17.430894 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:17.430865    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2281431c-56b4-475a-a7dc-35c1bbe04e1a-credential-socket" pod "2281431c-56b4-475a-a7dc-35c1bbe04e1a" (UID: "2281431c-56b4-475a-a7dc-35c1bbe04e1a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:50:17.430973 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:17.430939    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2281431c-56b4-475a-a7dc-35c1bbe04e1a-istiod-ca-cert" pod "2281431c-56b4-475a-a7dc-35c1bbe04e1a" (UID: "2281431c-56b4-475a-a7dc-35c1bbe04e1a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:50:17.431190 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:17.431154    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2281431c-56b4-475a-a7dc-35c1bbe04e1a-workload-certs" pod "2281431c-56b4-475a-a7dc-35c1bbe04e1a" (UID: "2281431c-56b4-475a-a7dc-35c1bbe04e1a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:50:17.431283 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:17.431218    4477 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2281431c-56b4-475a-a7dc-35c1bbe04e1a-workload-socket\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:50:17.431283 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:17.431239    4477 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2281431c-56b4-475a-a7dc-35c1bbe04e1a-istiod-ca-cert\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:50:17.431283 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:17.431253    4477 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2281431c-56b4-475a-a7dc-35c1bbe04e1a-credential-socket\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:50:17.431283 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:17.431266    4477 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2281431c-56b4-475a-a7dc-35c1bbe04e1a-istio-data\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:50:17.433062 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:17.432980    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2281431c-56b4-475a-a7dc-35c1bbe04e1a-istio-podinfo" pod "2281431c-56b4-475a-a7dc-35c1bbe04e1a" (UID: "2281431c-56b4-475a-a7dc-35c1bbe04e1a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:50:17.433062 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:17.432978    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2281431c-56b4-475a-a7dc-35c1bbe04e1a-istio-envoy" pod "2281431c-56b4-475a-a7dc-35c1bbe04e1a" (UID: "2281431c-56b4-475a-a7dc-35c1bbe04e1a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:50:17.433186 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:17.433106    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2281431c-56b4-475a-a7dc-35c1bbe04e1a-kube-api-access-dwqnl" pod "2281431c-56b4-475a-a7dc-35c1bbe04e1a" (UID: "2281431c-56b4-475a-a7dc-35c1bbe04e1a"). InnerVolumeSpecName "kube-api-access-dwqnl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:50:17.433186 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:17.433131    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2281431c-56b4-475a-a7dc-35c1bbe04e1a-istio-token" pod "2281431c-56b4-475a-a7dc-35c1bbe04e1a" (UID: "2281431c-56b4-475a-a7dc-35c1bbe04e1a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:50:17.532541 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:17.532476    4477 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2281431c-56b4-475a-a7dc-35c1bbe04e1a-istio-podinfo\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:50:17.532541 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:17.532509    4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwqnl\" (UniqueName: \"kubernetes.io/projected/2281431c-56b4-475a-a7dc-35c1bbe04e1a-kube-api-access-dwqnl\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:50:17.532541 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:17.532520    4477 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2281431c-56b4-475a-a7dc-35c1bbe04e1a-istio-envoy\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:50:17.532541 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:17.532529    4477 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2281431c-56b4-475a-a7dc-35c1bbe04e1a-workload-certs\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:50:17.532541 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:17.532538    4477 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2281431c-56b4-475a-a7dc-35c1bbe04e1a-istio-token\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:50:18.175683 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:18.175651    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-znd9z"
Oct 30 22:50:18.176189 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:18.175695    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-znd9z" event={"ID":"2281431c-56b4-475a-a7dc-35c1bbe04e1a","Type":"ContainerDied","Data":"28322b7823ca3b653837b0ef429897879a0aa171ea1a3f2ce72c97ed04e8df23"}
Oct 30 22:50:18.176189 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:18.175761    4477 scope.go:122] "RemoveContainer" containerID="d0afc0d62e8f97d2464f010d461c8190334728789c0ece70c2b3a208ee4fbbfb"
Oct 30 22:50:18.239366 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:18.239340    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-znd9z"]
Oct 30 22:50:18.244550 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:18.244529    4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-867f12-openshift-default-6584f84965-znd9z"]
Oct 30 22:50:20.193717 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:20.193679    4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2281431c-56b4-475a-a7dc-35c1bbe04e1a" path="/var/lib/kubelet/pods/2281431c-56b4-475a-a7dc-35c1bbe04e1a/volumes"
Oct 30 22:50:23.777986 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:23.777953    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6bz92"]
Oct 30 22:50:24.056104 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:24.056017    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-23aab2-55d6dff57-kp9bb"]
Oct 30 22:50:24.056555 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:24.056542    4477 cpu_manager.go:414] "containerMap: removing container" podUID="5ff74f16-8919-4c9c-9b49-f333618cdf65" containerName="maas-api"
Oct 30 22:50:24.056606 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:24.056557    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ff74f16-8919-4c9c-9b49-f333618cdf65" containerName="maas-api"
Oct 30 22:50:24.056606 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:24.056590    4477 cpu_manager.go:414] "containerMap: removing container" podUID="2281431c-56b4-475a-a7dc-35c1bbe04e1a" containerName="istio-proxy"
Oct 30 22:50:24.056606 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:24.056595    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2281431c-56b4-475a-a7dc-35c1bbe04e1a" containerName="istio-proxy"
Oct 30 22:50:24.056708 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:24.056685    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="2281431c-56b4-475a-a7dc-35c1bbe04e1a" containerName="istio-proxy"
Oct 30 22:50:24.056708 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:24.056698    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ff74f16-8919-4c9c-9b49-f333618cdf65" containerName="maas-api"
Oct 30 22:50:24.061725 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:24.061704    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-23aab2-55d6dff57-kp9bb"
Oct 30 22:50:24.088292 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:24.088273    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-23aab2-serving-cert\""
Oct 30 22:50:24.098529 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:24.098505    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-23aab2-55d6dff57-kp9bb"]
Oct 30 22:50:24.108371 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:24.108343    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kflw7\" (UniqueName: \"kubernetes.io/projected/d12a3703-b587-4f7b-9769-d4322c840eeb-kube-api-access-kflw7\") pod \"maas-api-e2e-shared-b-w1-23aab2-55d6dff57-kp9bb\" (UID: \"d12a3703-b587-4f7b-9769-d4322c840eeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-23aab2-55d6dff57-kp9bb"
Oct 30 22:50:24.108502 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:24.108405    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d12a3703-b587-4f7b-9769-d4322c840eeb-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-23aab2-55d6dff57-kp9bb\" (UID: \"d12a3703-b587-4f7b-9769-d4322c840eeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-23aab2-55d6dff57-kp9bb"
Oct 30 22:50:24.108565 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:24.108535    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d12a3703-b587-4f7b-9769-d4322c840eeb-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-23aab2-55d6dff57-kp9bb\" (UID: \"d12a3703-b587-4f7b-9769-d4322c840eeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-23aab2-55d6dff57-kp9bb"
Oct 30 22:50:24.108755 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:24.108733    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d12a3703-b587-4f7b-9769-d4322c840eeb-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-23aab2-55d6dff57-kp9bb\" (UID: \"d12a3703-b587-4f7b-9769-d4322c840eeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-23aab2-55d6dff57-kp9bb"
Oct 30 22:50:24.209549 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:24.209516    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d12a3703-b587-4f7b-9769-d4322c840eeb-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-23aab2-55d6dff57-kp9bb\" (UID: \"d12a3703-b587-4f7b-9769-d4322c840eeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-23aab2-55d6dff57-kp9bb"
Oct 30 22:50:24.209696 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:24.209637    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d12a3703-b587-4f7b-9769-d4322c840eeb-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-23aab2-55d6dff57-kp9bb\" (UID: \"d12a3703-b587-4f7b-9769-d4322c840eeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-23aab2-55d6dff57-kp9bb"
Oct 30 22:50:24.209696 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:24.209689    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kflw7\" (UniqueName: \"kubernetes.io/projected/d12a3703-b587-4f7b-9769-d4322c840eeb-kube-api-access-kflw7\") pod \"maas-api-e2e-shared-b-w1-23aab2-55d6dff57-kp9bb\" (UID: \"d12a3703-b587-4f7b-9769-d4322c840eeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-23aab2-55d6dff57-kp9bb"
Oct 30 22:50:24.209824 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:24.209735    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d12a3703-b587-4f7b-9769-d4322c840eeb-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-23aab2-55d6dff57-kp9bb\" (UID: \"d12a3703-b587-4f7b-9769-d4322c840eeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-23aab2-55d6dff57-kp9bb"
Oct 30 22:50:24.209889 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:50:24.209840    4477 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-23aab2-serving-cert: secret "maas-api-e2e-shared-b-w1-23aab2-serving-cert" not found
Oct 30 22:50:24.209949 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:50:24.209908    4477 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d12a3703-b587-4f7b-9769-d4322c840eeb-maas-api-tls podName:d12a3703-b587-4f7b-9769-d4322c840eeb nodeName:}" failed. No retries permitted until 2026-10-30 22:50:24.709886702 +0000 UTC m=+2608.635812902 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d12a3703-b587-4f7b-9769-d4322c840eeb-maas-api-tls") pod "maas-api-e2e-shared-b-w1-23aab2-55d6dff57-kp9bb" (UID: "d12a3703-b587-4f7b-9769-d4322c840eeb") : secret "maas-api-e2e-shared-b-w1-23aab2-serving-cert" not found
Oct 30 22:50:24.210347 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:24.210323    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d12a3703-b587-4f7b-9769-d4322c840eeb-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-23aab2-55d6dff57-kp9bb\" (UID: \"d12a3703-b587-4f7b-9769-d4322c840eeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-23aab2-55d6dff57-kp9bb"
Oct 30 22:50:24.212659 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:24.212635    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d12a3703-b587-4f7b-9769-d4322c840eeb-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-23aab2-55d6dff57-kp9bb\" (UID: \"d12a3703-b587-4f7b-9769-d4322c840eeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-23aab2-55d6dff57-kp9bb"
Oct 30 22:50:24.243308 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:24.243281    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kflw7\" (UniqueName: \"kubernetes.io/projected/d12a3703-b587-4f7b-9769-d4322c840eeb-kube-api-access-kflw7\") pod \"maas-api-e2e-shared-b-w1-23aab2-55d6dff57-kp9bb\" (UID: \"d12a3703-b587-4f7b-9769-d4322c840eeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-23aab2-55d6dff57-kp9bb"
Oct 30 22:50:24.715172 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:24.715139    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d12a3703-b587-4f7b-9769-d4322c840eeb-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-23aab2-55d6dff57-kp9bb\" (UID: \"d12a3703-b587-4f7b-9769-d4322c840eeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-23aab2-55d6dff57-kp9bb"
Oct 30 22:50:24.717596 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:24.717569    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d12a3703-b587-4f7b-9769-d4322c840eeb-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-23aab2-55d6dff57-kp9bb\" (UID: \"d12a3703-b587-4f7b-9769-d4322c840eeb\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-23aab2-55d6dff57-kp9bb"
Oct 30 22:50:24.974985 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:24.974917    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-23aab2-55d6dff57-kp9bb"
Oct 30 22:50:25.307167 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:25.307098    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-23aab2-55d6dff57-kp9bb"]
Oct 30 22:50:25.311718 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:50:25.311683    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd12a3703_b587_4f7b_9769_d4322c840eeb.slice/crio-8ff918e43f5c229eb8b66cbd1de36ff13794b840e679f57191d9af4b4212b5e7 WatchSource:0}: Error finding container 8ff918e43f5c229eb8b66cbd1de36ff13794b840e679f57191d9af4b4212b5e7: Status 404 returned error can't find the container with id 8ff918e43f5c229eb8b66cbd1de36ff13794b840e679f57191d9af4b4212b5e7
Oct 30 22:50:25.313260 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:25.313237    4477 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 22:50:26.226642 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:26.226603    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-23aab2-55d6dff57-kp9bb" event={"ID":"d12a3703-b587-4f7b-9769-d4322c840eeb","Type":"ContainerStarted","Data":"7e663b22a0e1326e4ac321d868a9e74d178016efe5d4cdd09d3b36f048414148"}
Oct 30 22:50:26.227089 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:26.226670    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-23aab2-55d6dff57-kp9bb" event={"ID":"d12a3703-b587-4f7b-9769-d4322c840eeb","Type":"ContainerStarted","Data":"8ff918e43f5c229eb8b66cbd1de36ff13794b840e679f57191d9af4b4212b5e7"}
Oct 30 22:50:26.227089 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:26.226703    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-23aab2-55d6dff57-kp9bb"
Oct 30 22:50:26.266708 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:26.266614    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-23aab2-55d6dff57-kp9bb" podStartSLOduration=1.896851528 podStartE2EDuration="2.266589872s" totalImagesPullingTime="369.738344ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:50:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:50:26.265293582 +0000 UTC m=+2610.191219780" watchObservedRunningTime="2026-10-30 22:50:26.266589872 +0000 UTC m=+2610.192516072"
Oct 30 22:50:26.283417 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:50:26.283381    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 22:50:27.087731 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:27.087696    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-80ecb3/sub-test-model-80ecb3-kserve-9b4b44659-vnh9d"]
Oct 30 22:50:27.092112 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:27.092086    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-80ecb3/sub-test-model-80ecb3-kserve-9b4b44659-vnh9d"
Oct 30 22:50:27.112558 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:27.112531    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-80ecb3/sub-test-model-80ecb3-kserve-9b4b44659-vnh9d"]
Oct 30 22:50:27.120898 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:27.120870    4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-80ecb3\"/\"openshift-service-ca.crt\""
Oct 30 22:50:27.121022 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:27.120912    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-80ecb3\"/\"default-dockercfg-5kfq4\""
Oct 30 22:50:27.121022 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:27.120994    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-80ecb3\"/\"sub-test-model-80ecb3-kserve-self-signed-certs\""
Oct 30 22:50:27.121273 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:27.121101    4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-80ecb3\"/\"kube-root-ca.crt\""
Oct 30 22:50:27.161948 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:27.161920    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c6d7321-d1ee-4023-ae2e-82dc126de49d-tmp-dir\") pod \"sub-test-model-80ecb3-kserve-9b4b44659-vnh9d\" (UID: \"3c6d7321-d1ee-4023-ae2e-82dc126de49d\") " pod="ai-tenant-e2e-shared-a-w2-80ecb3/sub-test-model-80ecb3-kserve-9b4b44659-vnh9d"
Oct 30 22:50:27.162109 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:27.161971    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c6d7321-d1ee-4023-ae2e-82dc126de49d-model-cache\") pod \"sub-test-model-80ecb3-kserve-9b4b44659-vnh9d\" (UID: \"3c6d7321-d1ee-4023-ae2e-82dc126de49d\") " pod="ai-tenant-e2e-shared-a-w2-80ecb3/sub-test-model-80ecb3-kserve-9b4b44659-vnh9d"
Oct 30 22:50:27.162179 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:27.162096    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c6d7321-d1ee-4023-ae2e-82dc126de49d-tls-certs\") pod \"sub-test-model-80ecb3-kserve-9b4b44659-vnh9d\" (UID: \"3c6d7321-d1ee-4023-ae2e-82dc126de49d\") " pod="ai-tenant-e2e-shared-a-w2-80ecb3/sub-test-model-80ecb3-kserve-9b4b44659-vnh9d"
Oct 30 22:50:27.162243 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:27.162194    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c6d7321-d1ee-4023-ae2e-82dc126de49d-home\") pod \"sub-test-model-80ecb3-kserve-9b4b44659-vnh9d\" (UID: \"3c6d7321-d1ee-4023-ae2e-82dc126de49d\") " pod="ai-tenant-e2e-shared-a-w2-80ecb3/sub-test-model-80ecb3-kserve-9b4b44659-vnh9d"
Oct 30 22:50:27.162302 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:27.162276    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c6d7321-d1ee-4023-ae2e-82dc126de49d-dshm\") pod \"sub-test-model-80ecb3-kserve-9b4b44659-vnh9d\" (UID: \"3c6d7321-d1ee-4023-ae2e-82dc126de49d\") " pod="ai-tenant-e2e-shared-a-w2-80ecb3/sub-test-model-80ecb3-kserve-9b4b44659-vnh9d"
Oct 30 22:50:27.162410 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:27.162393    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9rtb\" (UniqueName: \"kubernetes.io/projected/3c6d7321-d1ee-4023-ae2e-82dc126de49d-kube-api-access-w9rtb\") pod \"sub-test-model-80ecb3-kserve-9b4b44659-vnh9d\" (UID: \"3c6d7321-d1ee-4023-ae2e-82dc126de49d\") " pod="ai-tenant-e2e-shared-a-w2-80ecb3/sub-test-model-80ecb3-kserve-9b4b44659-vnh9d"
Oct 30 22:50:27.263720 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:27.263685    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c6d7321-d1ee-4023-ae2e-82dc126de49d-model-cache\") pod \"sub-test-model-80ecb3-kserve-9b4b44659-vnh9d\" (UID: \"3c6d7321-d1ee-4023-ae2e-82dc126de49d\") " pod="ai-tenant-e2e-shared-a-w2-80ecb3/sub-test-model-80ecb3-kserve-9b4b44659-vnh9d"
Oct 30 22:50:27.264223 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:27.263758    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c6d7321-d1ee-4023-ae2e-82dc126de49d-tls-certs\") pod \"sub-test-model-80ecb3-kserve-9b4b44659-vnh9d\" (UID: \"3c6d7321-d1ee-4023-ae2e-82dc126de49d\") " pod="ai-tenant-e2e-shared-a-w2-80ecb3/sub-test-model-80ecb3-kserve-9b4b44659-vnh9d"
Oct 30 22:50:27.264223 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:27.263845    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c6d7321-d1ee-4023-ae2e-82dc126de49d-home\") pod \"sub-test-model-80ecb3-kserve-9b4b44659-vnh9d\" (UID: \"3c6d7321-d1ee-4023-ae2e-82dc126de49d\") " pod="ai-tenant-e2e-shared-a-w2-80ecb3/sub-test-model-80ecb3-kserve-9b4b44659-vnh9d"
Oct 30 22:50:27.264223 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:27.264068    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c6d7321-d1ee-4023-ae2e-82dc126de49d-dshm\") pod \"sub-test-model-80ecb3-kserve-9b4b44659-vnh9d\" (UID: \"3c6d7321-d1ee-4023-ae2e-82dc126de49d\") " pod="ai-tenant-e2e-shared-a-w2-80ecb3/sub-test-model-80ecb3-kserve-9b4b44659-vnh9d"
Oct 30 22:50:27.264223 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:27.264173    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c6d7321-d1ee-4023-ae2e-82dc126de49d-model-cache\") pod \"sub-test-model-80ecb3-kserve-9b4b44659-vnh9d\" (UID: \"3c6d7321-d1ee-4023-ae2e-82dc126de49d\") " pod="ai-tenant-e2e-shared-a-w2-80ecb3/sub-test-model-80ecb3-kserve-9b4b44659-vnh9d"
Oct 30 22:50:27.264533 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:27.264248    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c6d7321-d1ee-4023-ae2e-82dc126de49d-home\") pod \"sub-test-model-80ecb3-kserve-9b4b44659-vnh9d\" (UID: \"3c6d7321-d1ee-4023-ae2e-82dc126de49d\") " pod="ai-tenant-e2e-shared-a-w2-80ecb3/sub-test-model-80ecb3-kserve-9b4b44659-vnh9d"
Oct 30 22:50:27.264533 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:27.264279    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9rtb\" (UniqueName: \"kubernetes.io/projected/3c6d7321-d1ee-4023-ae2e-82dc126de49d-kube-api-access-w9rtb\") pod \"sub-test-model-80ecb3-kserve-9b4b44659-vnh9d\" (UID: \"3c6d7321-d1ee-4023-ae2e-82dc126de49d\") " pod="ai-tenant-e2e-shared-a-w2-80ecb3/sub-test-model-80ecb3-kserve-9b4b44659-vnh9d"
Oct 30 22:50:27.264533 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:27.264335    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c6d7321-d1ee-4023-ae2e-82dc126de49d-tmp-dir\") pod \"sub-test-model-80ecb3-kserve-9b4b44659-vnh9d\" (UID: \"3c6d7321-d1ee-4023-ae2e-82dc126de49d\") " pod="ai-tenant-e2e-shared-a-w2-80ecb3/sub-test-model-80ecb3-kserve-9b4b44659-vnh9d"
Oct 30 22:50:27.265253 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:27.265228    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c6d7321-d1ee-4023-ae2e-82dc126de49d-tmp-dir\") pod \"sub-test-model-80ecb3-kserve-9b4b44659-vnh9d\" (UID: \"3c6d7321-d1ee-4023-ae2e-82dc126de49d\") " pod="ai-tenant-e2e-shared-a-w2-80ecb3/sub-test-model-80ecb3-kserve-9b4b44659-vnh9d"
Oct 30 22:50:27.266750 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:27.266723    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c6d7321-d1ee-4023-ae2e-82dc126de49d-dshm\") pod \"sub-test-model-80ecb3-kserve-9b4b44659-vnh9d\" (UID: \"3c6d7321-d1ee-4023-ae2e-82dc126de49d\") " pod="ai-tenant-e2e-shared-a-w2-80ecb3/sub-test-model-80ecb3-kserve-9b4b44659-vnh9d"
Oct 30 22:50:27.266855 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:27.266818    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c6d7321-d1ee-4023-ae2e-82dc126de49d-tls-certs\") pod \"sub-test-model-80ecb3-kserve-9b4b44659-vnh9d\" (UID: \"3c6d7321-d1ee-4023-ae2e-82dc126de49d\") " pod="ai-tenant-e2e-shared-a-w2-80ecb3/sub-test-model-80ecb3-kserve-9b4b44659-vnh9d"
Oct 30 22:50:27.283730 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:27.283708    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9rtb\" (UniqueName: \"kubernetes.io/projected/3c6d7321-d1ee-4023-ae2e-82dc126de49d-kube-api-access-w9rtb\") pod \"sub-test-model-80ecb3-kserve-9b4b44659-vnh9d\" (UID: \"3c6d7321-d1ee-4023-ae2e-82dc126de49d\") " pod="ai-tenant-e2e-shared-a-w2-80ecb3/sub-test-model-80ecb3-kserve-9b4b44659-vnh9d"
Oct 30 22:50:27.407813 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:27.407784    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-80ecb3/sub-test-model-80ecb3-kserve-9b4b44659-vnh9d"
Oct 30 22:50:27.729204 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:27.729127    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-80ecb3/sub-test-model-80ecb3-kserve-9b4b44659-vnh9d"]
Oct 30 22:50:27.733834 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:50:27.733789    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3c6d7321_d1ee_4023_ae2e_82dc126de49d.slice/crio-6140070c663f40a6509719594d51fa8e0f303a187fdb639c54019f2f7169a56d WatchSource:0}: Error finding container 6140070c663f40a6509719594d51fa8e0f303a187fdb639c54019f2f7169a56d: Status 404 returned error can't find the container with id 6140070c663f40a6509719594d51fa8e0f303a187fdb639c54019f2f7169a56d
Oct 30 22:50:28.243366 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:28.243325    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-80ecb3/sub-test-model-80ecb3-kserve-9b4b44659-vnh9d" event={"ID":"3c6d7321-d1ee-4023-ae2e-82dc126de49d","Type":"ContainerStarted","Data":"399622a958c8788964713dbee6025d9f1914f857c983cdfdd4719b4839ee7671"}
Oct 30 22:50:28.243565 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:28.243380    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-80ecb3/sub-test-model-80ecb3-kserve-9b4b44659-vnh9d" event={"ID":"3c6d7321-d1ee-4023-ae2e-82dc126de49d","Type":"ContainerStarted","Data":"6140070c663f40a6509719594d51fa8e0f303a187fdb639c54019f2f7169a56d"}
Oct 30 22:50:28.287392 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:28.287344    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-80ecb3/sub-test-model-80ecb3-kserve-9b4b44659-vnh9d" podStartSLOduration=1.287331713 podStartE2EDuration="1.287331713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:50:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:50:28.286606498 +0000 UTC m=+2612.212532725" watchObservedRunningTime="2026-10-30 22:50:28.287331713 +0000 UTC m=+2612.213257919"
Oct 30 22:50:28.408598 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:28.408253    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-80ecb3/sub-test-model-80ecb3-kserve-9b4b44659-vnh9d"
Oct 30 22:50:32.237075 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:32.237029    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-23aab2-55d6dff57-kp9bb"
Oct 30 22:50:37.408380 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:37.408344    4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-80ecb3/sub-test-model-80ecb3-kserve-9b4b44659-vnh9d"
Oct 30 22:50:37.421021 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:37.420992    4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-80ecb3/sub-test-model-80ecb3-kserve-9b4b44659-vnh9d"
Oct 30 22:50:38.314089 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:38.314062    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-80ecb3/sub-test-model-80ecb3-kserve-9b4b44659-vnh9d"
Oct 30 22:50:52.112303 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:52.112272    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cdc122-65f9cc65b8-95mpc"]
Oct 30 22:50:52.116666 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:52.116647    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cdc122-65f9cc65b8-95mpc"
Oct 30 22:50:52.131186 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:52.131167    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-cdc122-serving-cert\""
Oct 30 22:50:52.145832 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:52.145808    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cdc122-65f9cc65b8-95mpc"]
Oct 30 22:50:52.151147 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:52.151120    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec4328f4-72a8-4462-b87c-66415cbc6989-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-cdc122-65f9cc65b8-95mpc\" (UID: \"ec4328f4-72a8-4462-b87c-66415cbc6989\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cdc122-65f9cc65b8-95mpc"
Oct 30 22:50:52.151282 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:52.151264    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec4328f4-72a8-4462-b87c-66415cbc6989-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-cdc122-65f9cc65b8-95mpc\" (UID: \"ec4328f4-72a8-4462-b87c-66415cbc6989\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cdc122-65f9cc65b8-95mpc"
Oct 30 22:50:52.151352 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:52.151321    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sghrh\" (UniqueName: \"kubernetes.io/projected/ec4328f4-72a8-4462-b87c-66415cbc6989-kube-api-access-sghrh\") pod \"maas-api-e2e-shared-a-w3-cdc122-65f9cc65b8-95mpc\" (UID: \"ec4328f4-72a8-4462-b87c-66415cbc6989\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cdc122-65f9cc65b8-95mpc"
Oct 30 22:50:52.151428 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:52.151361    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec4328f4-72a8-4462-b87c-66415cbc6989-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-cdc122-65f9cc65b8-95mpc\" (UID: \"ec4328f4-72a8-4462-b87c-66415cbc6989\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cdc122-65f9cc65b8-95mpc"
Oct 30 22:50:52.252970 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:52.252933    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec4328f4-72a8-4462-b87c-66415cbc6989-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-cdc122-65f9cc65b8-95mpc\" (UID: \"ec4328f4-72a8-4462-b87c-66415cbc6989\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cdc122-65f9cc65b8-95mpc"
Oct 30 22:50:52.253176 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:52.253114    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec4328f4-72a8-4462-b87c-66415cbc6989-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-cdc122-65f9cc65b8-95mpc\" (UID: \"ec4328f4-72a8-4462-b87c-66415cbc6989\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cdc122-65f9cc65b8-95mpc"
Oct 30 22:50:52.253176 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:52.253161    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sghrh\" (UniqueName: \"kubernetes.io/projected/ec4328f4-72a8-4462-b87c-66415cbc6989-kube-api-access-sghrh\") pod \"maas-api-e2e-shared-a-w3-cdc122-65f9cc65b8-95mpc\" (UID: \"ec4328f4-72a8-4462-b87c-66415cbc6989\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cdc122-65f9cc65b8-95mpc"
Oct 30 22:50:52.253320 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:52.253196    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec4328f4-72a8-4462-b87c-66415cbc6989-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-cdc122-65f9cc65b8-95mpc\" (UID: \"ec4328f4-72a8-4462-b87c-66415cbc6989\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cdc122-65f9cc65b8-95mpc"
Oct 30 22:50:52.253849 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:52.253823    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec4328f4-72a8-4462-b87c-66415cbc6989-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-cdc122-65f9cc65b8-95mpc\" (UID: \"ec4328f4-72a8-4462-b87c-66415cbc6989\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cdc122-65f9cc65b8-95mpc"
Oct 30 22:50:52.255511 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:52.255494    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec4328f4-72a8-4462-b87c-66415cbc6989-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-cdc122-65f9cc65b8-95mpc\" (UID: \"ec4328f4-72a8-4462-b87c-66415cbc6989\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cdc122-65f9cc65b8-95mpc"
Oct 30 22:50:52.255574 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:52.255547    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec4328f4-72a8-4462-b87c-66415cbc6989-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-cdc122-65f9cc65b8-95mpc\" (UID: \"ec4328f4-72a8-4462-b87c-66415cbc6989\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cdc122-65f9cc65b8-95mpc"
Oct 30 22:50:52.271571 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:52.271550    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sghrh\" (UniqueName: \"kubernetes.io/projected/ec4328f4-72a8-4462-b87c-66415cbc6989-kube-api-access-sghrh\") pod \"maas-api-e2e-shared-a-w3-cdc122-65f9cc65b8-95mpc\" (UID: \"ec4328f4-72a8-4462-b87c-66415cbc6989\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cdc122-65f9cc65b8-95mpc"
Oct 30 22:50:52.432302 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:52.432262    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cdc122-65f9cc65b8-95mpc"
Oct 30 22:50:52.734327 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:52.734223    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cdc122-65f9cc65b8-95mpc"]
Oct 30 22:50:52.738242 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:50:52.738218    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec4328f4_72a8_4462_b87c_66415cbc6989.slice/crio-11081b1468b302a580a3d1fa2f7c6588854dd285672790bf33cf51ae12433d3b WatchSource:0}: Error finding container 11081b1468b302a580a3d1fa2f7c6588854dd285672790bf33cf51ae12433d3b: Status 404 returned error can't find the container with id 11081b1468b302a580a3d1fa2f7c6588854dd285672790bf33cf51ae12433d3b
Oct 30 22:50:53.397777 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:53.397733    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cdc122-65f9cc65b8-95mpc" event={"ID":"ec4328f4-72a8-4462-b87c-66415cbc6989","Type":"ContainerStarted","Data":"4ebd4aec5ad9ec3ab3f8b862a35aa855b44a8f142c0caeb8959a282d02b1ef82"}
Oct 30 22:50:53.398216 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:53.397799    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cdc122-65f9cc65b8-95mpc" event={"ID":"ec4328f4-72a8-4462-b87c-66415cbc6989","Type":"ContainerStarted","Data":"11081b1468b302a580a3d1fa2f7c6588854dd285672790bf33cf51ae12433d3b"}
Oct 30 22:50:53.398216 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:53.397924    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cdc122-65f9cc65b8-95mpc"
Oct 30 22:50:53.441474 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:53.441432    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cdc122-65f9cc65b8-95mpc" podStartSLOduration=0.97079958 podStartE2EDuration="1.441420666s" totalImagesPullingTime="470.621086ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:50:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:50:53.441358119 +0000 UTC m=+2637.367284339" watchObservedRunningTime="2026-10-30 22:50:53.441420666 +0000 UTC m=+2637.367346871"
Oct 30 22:50:56.268142 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:50:56.268108    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 22:50:59.408963 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:50:59.408936    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cdc122-65f9cc65b8-95mpc"
Oct 30 22:51:26.270899 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:51:26.270855    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 22:51:56.268065 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:51:56.268005    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 22:51:57.007965 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:51:57.007936    4477 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jcf8l_015f218d-bf1f-4e7e-bb5d-b805af003246/ovn-acl-logging/0.log"
Oct 30 22:51:57.039438 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:51:57.039415    4477 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jcf8l_015f218d-bf1f-4e7e-bb5d-b805af003246/ovn-acl-logging/0.log"
Oct 30 22:51:59.100200 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:51:59.100162    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e3a7dd-866fc6f4cb-gqb2f"]
Oct 30 22:51:59.104155 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:51:59.104133    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e3a7dd-866fc6f4cb-gqb2f"
Oct 30 22:51:59.121406 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:51:59.121364    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-e3a7dd-serving-cert\""
Oct 30 22:51:59.137565 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:51:59.137538    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e3a7dd-866fc6f4cb-gqb2f"]
Oct 30 22:51:59.234277 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:51:59.234248    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbb8248f-a7c8-4a23-80cd-ba28e98da140-metrics-certs\") pod \"maas-api-e2e-ipp-a-e3a7dd-866fc6f4cb-gqb2f\" (UID: \"dbb8248f-a7c8-4a23-80cd-ba28e98da140\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e3a7dd-866fc6f4cb-gqb2f"
Oct 30 22:51:59.234448 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:51:59.234429    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbb8248f-a7c8-4a23-80cd-ba28e98da140-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-e3a7dd-866fc6f4cb-gqb2f\" (UID: \"dbb8248f-a7c8-4a23-80cd-ba28e98da140\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e3a7dd-866fc6f4cb-gqb2f"
Oct 30 22:51:59.234516 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:51:59.234501    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dbb8248f-a7c8-4a23-80cd-ba28e98da140-maas-api-tls\") pod \"maas-api-e2e-ipp-a-e3a7dd-866fc6f4cb-gqb2f\" (UID: \"dbb8248f-a7c8-4a23-80cd-ba28e98da140\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e3a7dd-866fc6f4cb-gqb2f"
Oct 30 22:51:59.234567 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:51:59.234529    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtrtz\" (UniqueName: \"kubernetes.io/projected/dbb8248f-a7c8-4a23-80cd-ba28e98da140-kube-api-access-mtrtz\") pod \"maas-api-e2e-ipp-a-e3a7dd-866fc6f4cb-gqb2f\" (UID: \"dbb8248f-a7c8-4a23-80cd-ba28e98da140\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e3a7dd-866fc6f4cb-gqb2f"
Oct 30 22:51:59.337134 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:51:59.337094    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbb8248f-a7c8-4a23-80cd-ba28e98da140-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-e3a7dd-866fc6f4cb-gqb2f\" (UID: \"dbb8248f-a7c8-4a23-80cd-ba28e98da140\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e3a7dd-866fc6f4cb-gqb2f"
Oct 30 22:51:59.337579 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:51:59.337544    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dbb8248f-a7c8-4a23-80cd-ba28e98da140-maas-api-tls\") pod \"maas-api-e2e-ipp-a-e3a7dd-866fc6f4cb-gqb2f\" (UID: \"dbb8248f-a7c8-4a23-80cd-ba28e98da140\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e3a7dd-866fc6f4cb-gqb2f"
Oct 30 22:51:59.338722 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:51:59.337905    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtrtz\" (UniqueName: \"kubernetes.io/projected/dbb8248f-a7c8-4a23-80cd-ba28e98da140-kube-api-access-mtrtz\") pod \"maas-api-e2e-ipp-a-e3a7dd-866fc6f4cb-gqb2f\" (UID: \"dbb8248f-a7c8-4a23-80cd-ba28e98da140\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e3a7dd-866fc6f4cb-gqb2f"
Oct 30 22:51:59.338722 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:51:59.338157    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbb8248f-a7c8-4a23-80cd-ba28e98da140-metrics-certs\") pod \"maas-api-e2e-ipp-a-e3a7dd-866fc6f4cb-gqb2f\" (UID: \"dbb8248f-a7c8-4a23-80cd-ba28e98da140\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e3a7dd-866fc6f4cb-gqb2f"
Oct 30 22:51:59.341084 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:51:59.340273    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbb8248f-a7c8-4a23-80cd-ba28e98da140-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-e3a7dd-866fc6f4cb-gqb2f\" (UID: \"dbb8248f-a7c8-4a23-80cd-ba28e98da140\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e3a7dd-866fc6f4cb-gqb2f"
Oct 30 22:51:59.349281 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:51:59.349246    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dbb8248f-a7c8-4a23-80cd-ba28e98da140-maas-api-tls\") pod \"maas-api-e2e-ipp-a-e3a7dd-866fc6f4cb-gqb2f\" (UID: \"dbb8248f-a7c8-4a23-80cd-ba28e98da140\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e3a7dd-866fc6f4cb-gqb2f"
Oct 30 22:51:59.350115 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:51:59.350085    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbb8248f-a7c8-4a23-80cd-ba28e98da140-metrics-certs\") pod \"maas-api-e2e-ipp-a-e3a7dd-866fc6f4cb-gqb2f\" (UID: \"dbb8248f-a7c8-4a23-80cd-ba28e98da140\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e3a7dd-866fc6f4cb-gqb2f"
Oct 30 22:51:59.358561 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:51:59.358492    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtrtz\" (UniqueName: \"kubernetes.io/projected/dbb8248f-a7c8-4a23-80cd-ba28e98da140-kube-api-access-mtrtz\") pod \"maas-api-e2e-ipp-a-e3a7dd-866fc6f4cb-gqb2f\" (UID: \"dbb8248f-a7c8-4a23-80cd-ba28e98da140\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e3a7dd-866fc6f4cb-gqb2f"
Oct 30 22:51:59.424310 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:51:59.424285    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e3a7dd-866fc6f4cb-gqb2f"
Oct 30 22:51:59.722949 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:51:59.722787    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e3a7dd-866fc6f4cb-gqb2f"]
Oct 30 22:51:59.726328 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:51:59.726300    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbb8248f_a7c8_4a23_80cd_ba28e98da140.slice/crio-0d95721d320fd5170d74571d0879a891ac71c13bc5094231a118e1b906655554 WatchSource:0}: Error finding container 0d95721d320fd5170d74571d0879a891ac71c13bc5094231a118e1b906655554: Status 404 returned error can't find the container with id 0d95721d320fd5170d74571d0879a891ac71c13bc5094231a118e1b906655554
Oct 30 22:51:59.808100 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:51:59.808071    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e3a7dd-866fc6f4cb-gqb2f" event={"ID":"dbb8248f-a7c8-4a23-80cd-ba28e98da140","Type":"ContainerStarted","Data":"0d95721d320fd5170d74571d0879a891ac71c13bc5094231a118e1b906655554"}
Oct 30 22:52:00.820338 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:52:00.820293    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e3a7dd-866fc6f4cb-gqb2f" event={"ID":"dbb8248f-a7c8-4a23-80cd-ba28e98da140","Type":"ContainerStarted","Data":"272fddee0f30375f296ce1a617eb22a0752559e957bf069cf62e04e9ed249685"}
Oct 30 22:52:00.820790 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:52:00.820429    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e3a7dd-866fc6f4cb-gqb2f"
Oct 30 22:52:00.865327 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:52:00.865270    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e3a7dd-866fc6f4cb-gqb2f" podStartSLOduration=1.467851145 podStartE2EDuration="1.865256813s" totalImagesPullingTime="397.405668ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:51:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:52:00.863276562 +0000 UTC m=+2704.789202769" watchObservedRunningTime="2026-10-30 22:52:00.865256813 +0000 UTC m=+2704.791183050"
Oct 30 22:52:01.090079 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:52:01.089976    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w1-23aab2/auth-test-model-23aab2-kserve-7f6f98db57-rbmt7"]
Oct 30 22:52:01.093918 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:52:01.093891    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-23aab2/auth-test-model-23aab2-kserve-7f6f98db57-rbmt7"
Oct 30 22:52:01.109247 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:52:01.109226    4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-23aab2\"/\"openshift-service-ca.crt\""
Oct 30 22:52:01.109362 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:52:01.109266    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-23aab2\"/\"auth-test-model-23aab2-kserve-self-signed-certs\""
Oct 30 22:52:01.109362 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:52:01.109287    4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-23aab2\"/\"kube-root-ca.crt\""
Oct 30 22:52:01.109362 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:52:01.109329    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-23aab2\"/\"default-dockercfg-xtg2q\""
Oct 30 22:52:01.112821 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:52:01.112799    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-23aab2/auth-test-model-23aab2-kserve-7f6f98db57-rbmt7"]
Oct 30 22:52:01.171091 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:52:01.171033    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvfrq\" (UniqueName: \"kubernetes.io/projected/10316cee-ad81-4b83-acb8-6124dec5f00a-kube-api-access-rvfrq\") pod \"auth-test-model-23aab2-kserve-7f6f98db57-rbmt7\" (UID: \"10316cee-ad81-4b83-acb8-6124dec5f00a\") " pod="ai-tenant-e2e-shared-b-w1-23aab2/auth-test-model-23aab2-kserve-7f6f98db57-rbmt7"
Oct 30 22:52:01.171240 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:52:01.171104    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10316cee-ad81-4b83-acb8-6124dec5f00a-model-cache\") pod \"auth-test-model-23aab2-kserve-7f6f98db57-rbmt7\" (UID: \"10316cee-ad81-4b83-acb8-6124dec5f00a\") " pod="ai-tenant-e2e-shared-b-w1-23aab2/auth-test-model-23aab2-kserve-7f6f98db57-rbmt7"
Oct 30 22:52:01.171240 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:52:01.171199    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10316cee-ad81-4b83-acb8-6124dec5f00a-tmp-dir\") pod \"auth-test-model-23aab2-kserve-7f6f98db57-rbmt7\" (UID: \"10316cee-ad81-4b83-acb8-6124dec5f00a\") " pod="ai-tenant-e2e-shared-b-w1-23aab2/auth-test-model-23aab2-kserve-7f6f98db57-rbmt7"
Oct 30 22:52:01.171336 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:52:01.171249    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10316cee-ad81-4b83-acb8-6124dec5f00a-dshm\") pod \"auth-test-model-23aab2-kserve-7f6f98db57-rbmt7\" (UID: \"10316cee-ad81-4b83-acb8-6124dec5f00a\") " pod="ai-tenant-e2e-shared-b-w1-23aab2/auth-test-model-23aab2-kserve-7f6f98db57-rbmt7"
Oct 30 22:52:01.171383 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:52:01.171366    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10316cee-ad81-4b83-acb8-6124dec5f00a-home\") pod \"auth-test-model-23aab2-kserve-7f6f98db57-rbmt7\" (UID: \"10316cee-ad81-4b83-acb8-6124dec5f00a\") " pod="ai-tenant-e2e-shared-b-w1-23aab2/auth-test-model-23aab2-kserve-7f6f98db57-rbmt7"
Oct 30 22:52:01.171419 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:52:01.171404    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10316cee-ad81-4b83-acb8-6124dec5f00a-tls-certs\") pod \"auth-test-model-23aab2-kserve-7f6f98db57-rbmt7\" (UID: \"10316cee-ad81-4b83-acb8-6124dec5f00a\") " pod="ai-tenant-e2e-shared-b-w1-23aab2/auth-test-model-23aab2-kserve-7f6f98db57-rbmt7"
Oct 30 22:52:01.272623 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:52:01.272582    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10316cee-ad81-4b83-acb8-6124dec5f00a-home\") pod \"auth-test-model-23aab2-kserve-7f6f98db57-rbmt7\" (UID: \"10316cee-ad81-4b83-acb8-6124dec5f00a\") " pod="ai-tenant-e2e-shared-b-w1-23aab2/auth-test-model-23aab2-kserve-7f6f98db57-rbmt7"
Oct 30 22:52:01.272824 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:52:01.272642    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10316cee-ad81-4b83-acb8-6124dec5f00a-tls-certs\") pod \"auth-test-model-23aab2-kserve-7f6f98db57-rbmt7\" (UID: \"10316cee-ad81-4b83-acb8-6124dec5f00a\") " pod="ai-tenant-e2e-shared-b-w1-23aab2/auth-test-model-23aab2-kserve-7f6f98db57-rbmt7"
Oct 30 22:52:01.272824 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:52:01.272810    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvfrq\" (UniqueName: \"kubernetes.io/projected/10316cee-ad81-4b83-acb8-6124dec5f00a-kube-api-access-rvfrq\") pod \"auth-test-model-23aab2-kserve-7f6f98db57-rbmt7\" (UID: \"10316cee-ad81-4b83-acb8-6124dec5f00a\") " pod="ai-tenant-e2e-shared-b-w1-23aab2/auth-test-model-23aab2-kserve-7f6f98db57-rbmt7"
Oct 30 22:52:01.272952 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:52:01.272843    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10316cee-ad81-4b83-acb8-6124dec5f00a-model-cache\") pod \"auth-test-model-23aab2-kserve-7f6f98db57-rbmt7\" (UID: \"10316cee-ad81-4b83-acb8-6124dec5f00a\") " pod="ai-tenant-e2e-shared-b-w1-23aab2/auth-test-model-23aab2-kserve-7f6f98db57-rbmt7"
Oct 30 22:52:01.272952 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:52:01.272893    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10316cee-ad81-4b83-acb8-6124dec5f00a-tmp-dir\") pod \"auth-test-model-23aab2-kserve-7f6f98db57-rbmt7\" (UID: \"10316cee-ad81-4b83-acb8-6124dec5f00a\") " pod="ai-tenant-e2e-shared-b-w1-23aab2/auth-test-model-23aab2-kserve-7f6f98db57-rbmt7"
Oct 30 22:52:01.272952 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:52:01.272928    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10316cee-ad81-4b83-acb8-6124dec5f00a-dshm\") pod \"auth-test-model-23aab2-kserve-7f6f98db57-rbmt7\" (UID: \"10316cee-ad81-4b83-acb8-6124dec5f00a\") " pod="ai-tenant-e2e-shared-b-w1-23aab2/auth-test-model-23aab2-kserve-7f6f98db57-rbmt7"
Oct 30 22:52:01.273133 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:52:01.273114    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10316cee-ad81-4b83-acb8-6124dec5f00a-home\") pod \"auth-test-model-23aab2-kserve-7f6f98db57-rbmt7\" (UID: \"10316cee-ad81-4b83-acb8-6124dec5f00a\") " pod="ai-tenant-e2e-shared-b-w1-23aab2/auth-test-model-23aab2-kserve-7f6f98db57-rbmt7"
Oct 30 22:52:01.273231 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:52:01.273200    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10316cee-ad81-4b83-acb8-6124dec5f00a-model-cache\") pod \"auth-test-model-23aab2-kserve-7f6f98db57-rbmt7\" (UID: \"10316cee-ad81-4b83-acb8-6124dec5f00a\") " pod="ai-tenant-e2e-shared-b-w1-23aab2/auth-test-model-23aab2-kserve-7f6f98db57-rbmt7"
Oct 30 22:52:01.273435 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:52:01.273414    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10316cee-ad81-4b83-acb8-6124dec5f00a-tmp-dir\") pod \"auth-test-model-23aab2-kserve-7f6f98db57-rbmt7\" (UID: \"10316cee-ad81-4b83-acb8-6124dec5f00a\") " pod="ai-tenant-e2e-shared-b-w1-23aab2/auth-test-model-23aab2-kserve-7f6f98db57-rbmt7"
Oct 30 22:52:01.275284 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:52:01.275257    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10316cee-ad81-4b83-acb8-6124dec5f00a-dshm\") pod \"auth-test-model-23aab2-kserve-7f6f98db57-rbmt7\" (UID: \"10316cee-ad81-4b83-acb8-6124dec5f00a\") " pod="ai-tenant-e2e-shared-b-w1-23aab2/auth-test-model-23aab2-kserve-7f6f98db57-rbmt7"
Oct 30 22:52:01.275874 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:52:01.275852    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10316cee-ad81-4b83-acb8-6124dec5f00a-tls-certs\") pod \"auth-test-model-23aab2-kserve-7f6f98db57-rbmt7\" (UID: \"10316cee-ad81-4b83-acb8-6124dec5f00a\") " pod="ai-tenant-e2e-shared-b-w1-23aab2/auth-test-model-23aab2-kserve-7f6f98db57-rbmt7"
Oct 30 22:52:01.297422 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:52:01.297395    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvfrq\" (UniqueName: \"kubernetes.io/projected/10316cee-ad81-4b83-acb8-6124dec5f00a-kube-api-access-rvfrq\") pod \"auth-test-model-23aab2-kserve-7f6f98db57-rbmt7\" (UID: \"10316cee-ad81-4b83-acb8-6124dec5f00a\") " pod="ai-tenant-e2e-shared-b-w1-23aab2/auth-test-model-23aab2-kserve-7f6f98db57-rbmt7"
Oct 30 22:52:01.408795 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:52:01.408756    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-23aab2/auth-test-model-23aab2-kserve-7f6f98db57-rbmt7"
Oct 30 22:52:01.726097 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:52:01.725991    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-23aab2/auth-test-model-23aab2-kserve-7f6f98db57-rbmt7"]
Oct 30 22:52:01.730971 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:52:01.730940    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod10316cee_ad81_4b83_acb8_6124dec5f00a.slice/crio-e16171fd20876dfae636cf79d2dd3cd6b85f0cf76a633c074a0365eade19e8c4 WatchSource:0}: Error finding container e16171fd20876dfae636cf79d2dd3cd6b85f0cf76a633c074a0365eade19e8c4: Status 404 returned error can't find the container with id e16171fd20876dfae636cf79d2dd3cd6b85f0cf76a633c074a0365eade19e8c4
Oct 30 22:52:01.827167 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:52:01.827132    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-23aab2/auth-test-model-23aab2-kserve-7f6f98db57-rbmt7" event={"ID":"10316cee-ad81-4b83-acb8-6124dec5f00a","Type":"ContainerStarted","Data":"796329acc1656f87e03bb6e3d8e41370e72e651a4e22d16104a3cbe72e680afa"}
Oct 30 22:52:01.827558 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:52:01.827183    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-23aab2/auth-test-model-23aab2-kserve-7f6f98db57-rbmt7" event={"ID":"10316cee-ad81-4b83-acb8-6124dec5f00a","Type":"ContainerStarted","Data":"e16171fd20876dfae636cf79d2dd3cd6b85f0cf76a633c074a0365eade19e8c4"}
Oct 30 22:52:01.867720 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:52:01.867649    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w1-23aab2/auth-test-model-23aab2-kserve-7f6f98db57-rbmt7" podStartSLOduration=0.867635117 podStartE2EDuration="867.635117ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:52:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:52:01.865485519 +0000 UTC m=+2705.791411748" watchObservedRunningTime="2026-10-30 22:52:01.867635117 +0000 UTC m=+2705.793561322"
Oct 30 22:52:02.409442 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:52:02.409411    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-23aab2/auth-test-model-23aab2-kserve-7f6f98db57-rbmt7"
Oct 30 22:52:02.566195 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:52:02.566156    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6bz92"]
Oct 30 22:52:06.832892 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:52:06.832865    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e3a7dd-866fc6f4cb-gqb2f"
Oct 30 22:52:11.409229 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:52:11.409194    4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w1-23aab2/auth-test-model-23aab2-kserve-7f6f98db57-rbmt7"
Oct 30 22:52:11.423059 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:52:11.423019    4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w1-23aab2/auth-test-model-23aab2-kserve-7f6f98db57-rbmt7"
Oct 30 22:52:11.900268 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:52:11.900188    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-23aab2/auth-test-model-23aab2-kserve-7f6f98db57-rbmt7"
Oct 30 22:52:13.544761 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:52:13.544731    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-6bvgh"]
Oct 30 22:52:13.549320 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:52:13.549297    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-6bvgh"
Oct 30 22:52:13.572826 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:52:13.572794    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-6bvgh"]
Oct 30 22:52:13.607429 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:52:13.607396    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-cdc122-openshift-default-dockercfg-4sxwk\""
Oct 30 22:52:13.721309 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:52:13.721269    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18a561b5-289d-4401-8377-0be0130d658d-istio-envoy\") pod \"e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-6bvgh\" (UID: \"18a561b5-289d-4401-8377-0be0130d658d\") " pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-6bvgh"
Oct 30 22:52:13.721486 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:52:13.721391    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18a561b5-289d-4401-8377-0be0130d658d-workload-socket\") pod \"e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-6bvgh\" (UID: \"18a561b5-289d-4401-8377-0be0130d658d\") " pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-6bvgh"
Oct 30 22:52:13.721486 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:52:13.721453    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18a561b5-289d-4401-8377-0be0130d658d-istiod-ca-cert\") pod \"e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-6bvgh\" (UID: \"18a561b5-289d-4401-8377-0be0130d658d\") " pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-6bvgh"
Oct 30 22:52:13.721574 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:52:13.721502    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxbhf\" (UniqueName: \"kubernetes.io/projected/18a561b5-289d-4401-8377-0be0130d658d-kube-api-access-vxbhf\") pod \"e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-6bvgh\" (UID: \"18a561b5-289d-4401-8377-0be0130d658d\") " pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-6bvgh"
Oct 30 22:52:13.721636 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:52:13.721618    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18a561b5-289d-4401-8377-0be0130d658d-istio-token\") pod \"e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-6bvgh\" (UID: \"18a561b5-289d-4401-8377-0be0130d658d\") " pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-6bvgh"
Oct 30 22:52:13.721706 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:52:13.721694    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18a561b5-289d-4401-8377-0be0130d658d-istio-data\") pod \"e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-6bvgh\" (UID: \"18a561b5-289d-4401-8377-0be0130d658d\") " pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-6bvgh"
Oct 30 22:52:13.721744 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:52:13.721722    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18a561b5-289d-4401-8377-0be0130d658d-credential-socket\") pod \"e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-6bvgh\" (UID: \"18a561b5-289d-4401-8377-0be0130d658d\") " pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-6bvgh"
Oct 30 22:52:13.721744 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:52:13.721740    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18a561b5-289d-4401-8377-0be0130d658d-istio-podinfo\") pod \"e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-6bvgh\" (UID: \"18a561b5-289d-4401-8377-0be0130d658d\") " pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-6bvgh"
Oct 30 22:52:13.721871 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:52:13.721856    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18a561b5-289d-4401-8377-0be0130d658d-workload-certs\") pod \"e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-6bvgh\" (UID: \"18a561b5-289d-4401-8377-0be0130d658d\") " pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-6bvgh"
Oct 30 22:52:13.823414 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:52:13.823341    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18a561b5-289d-4401-8377-0be0130d658d-istio-envoy\") pod \"e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-6bvgh\" (UID: \"18a561b5-289d-4401-8377-0be0130d658d\") " pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-6bvgh"
Oct 30 22:52:13.823414 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:52:13.823395    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18a561b5-289d-4401-8377-0be0130d658d-workload-socket\") pod \"e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-6bvgh\" (UID: \"18a561b5-289d-4401-8377-0be0130d658d\") " pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-6bvgh"
Oct 30 22:52:13.823414 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:52:13.823412    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18a561b5-289d-4401-8377-0be0130d658d-istiod-ca-cert\") pod \"e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-6bvgh\" (UID: \"18a561b5-289d-4401-8377-0be0130d658d\") " pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-6bvgh"
Oct 30 22:52:13.823702 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:52:13.823444    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxbhf\" (UniqueName: \"kubernetes.io/projected/18a561b5-289d-4401-8377-0be0130d658d-kube-api-access-vxbhf\") pod \"e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-6bvgh\" (UID: \"18a561b5-289d-4401-8377-0be0130d658d\") " pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-6bvgh"
Oct 30 22:52:13.823702 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:52:13.823496    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18a561b5-289d-4401-8377-0be0130d658d-istio-token\") pod \"e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-6bvgh\" (UID: \"18a561b5-289d-4401-8377-0be0130d658d\") " pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-6bvgh"
Oct 30 22:52:13.823702 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:52:13.823519    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18a561b5-289d-4401-8377-0be0130d658d-istio-data\") pod \"e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-6bvgh\" (UID: \"18a561b5-289d-4401-8377-0be0130d658d\") " pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-6bvgh"
Oct 30 22:52:13.823702 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:52:13.823538    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18a561b5-289d-4401-8377-0be0130d658d-credential-socket\") pod \"e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-6bvgh\" (UID: \"18a561b5-289d-4401-8377-0be0130d658d\") " pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-6bvgh"
Oct 30 22:52:13.823702 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:52:13.823555    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18a561b5-289d-4401-8377-0be0130d658d-istio-podinfo\") pod \"e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-6bvgh\" (UID: \"18a561b5-289d-4401-8377-0be0130d658d\") " pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-6bvgh"
Oct 30 22:52:13.823702 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:52:13.823649    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18a561b5-289d-4401-8377-0be0130d658d-workload-certs\") pod \"e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-6bvgh\" (UID: \"18a561b5-289d-4401-8377-0be0130d658d\") " pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-6bvgh"
Oct 30 22:52:13.824019 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:52:13.823908    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18a561b5-289d-4401-8377-0be0130d658d-workload-socket\") pod \"e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-6bvgh\" (UID: \"18a561b5-289d-4401-8377-0be0130d658d\") " pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-6bvgh"
Oct 30 22:52:13.824019 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:52:13.823984    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18a561b5-289d-4401-8377-0be0130d658d-istio-data\") pod \"e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-6bvgh\" (UID: \"18a561b5-289d-4401-8377-0be0130d658d\") " pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-6bvgh"
Oct 30 22:52:13.824019 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:52:13.824014    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18a561b5-289d-4401-8377-0be0130d658d-workload-certs\") pod \"e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-6bvgh\" (UID: \"18a561b5-289d-4401-8377-0be0130d658d\") " pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-6bvgh"
Oct 30 22:52:13.824252 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:52:13.824229    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18a561b5-289d-4401-8377-0be0130d658d-credential-socket\") pod \"e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-6bvgh\" (UID: \"18a561b5-289d-4401-8377-0be0130d658d\") " pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-6bvgh"
Oct 30 22:52:13.824472 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:52:13.824452    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18a561b5-289d-4401-8377-0be0130d658d-istiod-ca-cert\") pod \"e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-6bvgh\" (UID: \"18a561b5-289d-4401-8377-0be0130d658d\") " pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-6bvgh"
Oct 30 22:52:13.825854 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:52:13.825830    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18a561b5-289d-4401-8377-0be0130d658d-istio-envoy\") pod \"e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-6bvgh\" (UID: \"18a561b5-289d-4401-8377-0be0130d658d\") " pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-6bvgh"
Oct 30 22:52:13.826278 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:52:13.826257    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18a561b5-289d-4401-8377-0be0130d658d-istio-podinfo\") pod \"e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-6bvgh\" (UID: \"18a561b5-289d-4401-8377-0be0130d658d\") " pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-6bvgh"
Oct 30 22:52:13.842521 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:52:13.842495    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxbhf\" (UniqueName: \"kubernetes.io/projected/18a561b5-289d-4401-8377-0be0130d658d-kube-api-access-vxbhf\") pod \"e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-6bvgh\" (UID: \"18a561b5-289d-4401-8377-0be0130d658d\") " pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-6bvgh"
Oct 30 22:52:13.842624 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:52:13.842525    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18a561b5-289d-4401-8377-0be0130d658d-istio-token\") pod \"e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-6bvgh\" (UID: \"18a561b5-289d-4401-8377-0be0130d658d\") " pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-6bvgh"
Oct 30 22:52:13.866215 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:52:13.866195    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-6bvgh"
Oct 30 22:52:14.190287 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:52:14.190262    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18a561b5_289d_4401_8377_0be0130d658d.slice/crio-4e916028dbaafb148f091bedc1b985d89be93d6e51025350f144ff53e34d83d0 WatchSource:0}: Error finding container 4e916028dbaafb148f091bedc1b985d89be93d6e51025350f144ff53e34d83d0: Status 404 returned error can't find the container with id 4e916028dbaafb148f091bedc1b985d89be93d6e51025350f144ff53e34d83d0
Oct 30 22:52:14.193408 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:52:14.193365    4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:52:14.193513 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:52:14.193442    4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:52:14.193513 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:52:14.193476    4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:52:14.195452 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:52:14.195434    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-6bvgh"]
Oct 30 22:52:14.917514 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:52:14.917480    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-6bvgh" event={"ID":"18a561b5-289d-4401-8377-0be0130d658d","Type":"ContainerStarted","Data":"4e1fa27a8e840f0dd8b7546ecdbfb0351b1b94e20a4a1801b43603755d4ee49d"}
Oct 30 22:52:14.917910 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:52:14.917539    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-6bvgh" event={"ID":"18a561b5-289d-4401-8377-0be0130d658d","Type":"ContainerStarted","Data":"4e916028dbaafb148f091bedc1b985d89be93d6e51025350f144ff53e34d83d0"}
Oct 30 22:52:15.019464 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:52:15.019412    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-6bvgh" podStartSLOduration=2.019398122 podStartE2EDuration="2.019398122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:52:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:52:14.991624408 +0000 UTC m=+2718.917550614" watchObservedRunningTime="2026-10-30 22:52:15.019398122 +0000 UTC m=+2718.945324327"
Oct 30 22:52:15.882067 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:52:15.874305    4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-6bvgh"
Oct 30 22:52:15.895086 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:52:15.894137    4477 patch_prober.go:29] interesting pod/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-6bvgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body=
Oct 30 22:52:15.895086 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:52:15.894193    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-6bvgh" podUID="18a561b5-289d-4401-8377-0be0130d658d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused"
Oct 30 22:52:16.866648 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:52:16.866610    4477 patch_prober.go:29] interesting pod/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-6bvgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body=
Oct 30 22:52:16.867064 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:52:16.866676    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-6bvgh" podUID="18a561b5-289d-4401-8377-0be0130d658d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused"
Oct 30 22:52:17.866984 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:52:17.866949    4477 patch_prober.go:29] interesting pod/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-6bvgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body=
Oct 30 22:52:17.867382 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:52:17.867008    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-6bvgh" podUID="18a561b5-289d-4401-8377-0be0130d658d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused"
Oct 30 22:52:18.866750 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:52:18.866712    4477 patch_prober.go:29] interesting pod/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-6bvgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body=
Oct 30 22:52:18.866945 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:52:18.866899    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-6bvgh" podUID="18a561b5-289d-4401-8377-0be0130d658d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused"
Oct 30 22:52:18.922604 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:52:18.922569    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e14e00-6bfcb896b7-xq4xg"]
Oct 30 22:52:18.927307 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:52:18.927287    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e14e00-6bfcb896b7-xq4xg"
Oct 30 22:52:18.946891 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:52:18.946866    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-e14e00-serving-cert\""
Oct 30 22:52:18.952408 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:52:18.952385    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e14e00-6bfcb896b7-xq4xg"]
Oct 30 22:52:19.006168 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:52:19.006132    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8157c822-ab10-4925-b3c9-6211dff26119-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-e14e00-6bfcb896b7-xq4xg\" (UID: \"8157c822-ab10-4925-b3c9-6211dff26119\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e14e00-6bfcb896b7-xq4xg"
Oct 30 22:52:19.006475 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:52:19.006440    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhvkt\" (UniqueName: \"kubernetes.io/projected/8157c822-ab10-4925-b3c9-6211dff26119-kube-api-access-fhvkt\") pod \"maas-api-e2e-shared-b-w3-e14e00-6bfcb896b7-xq4xg\" (UID: \"8157c822-ab10-4925-b3c9-6211dff26119\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e14e00-6bfcb896b7-xq4xg"
Oct 30 22:52:19.006475 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:52:19.006475    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8157c822-ab10-4925-b3c9-6211dff26119-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-e14e00-6bfcb896b7-xq4xg\" (UID: \"8157c822-ab10-4925-b3c9-6211dff26119\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e14e00-6bfcb896b7-xq4xg"
Oct 30 22:52:19.006671 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:52:19.006535    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8157c822-ab10-4925-b3c9-6211dff26119-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-e14e00-6bfcb896b7-xq4xg\" (UID: \"8157c822-ab10-4925-b3c9-6211dff26119\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e14e00-6bfcb896b7-xq4xg"
Oct 30 22:52:19.108239 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:52:19.108203    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8157c822-ab10-4925-b3c9-6211dff26119-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-e14e00-6bfcb896b7-xq4xg\" (UID: \"8157c822-ab10-4925-b3c9-6211dff26119\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e14e00-6bfcb896b7-xq4xg"
Oct 30 22:52:19.108409 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:52:19.108318    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhvkt\" (UniqueName: \"kubernetes.io/projected/8157c822-ab10-4925-b3c9-6211dff26119-kube-api-access-fhvkt\") pod \"maas-api-e2e-shared-b-w3-e14e00-6bfcb896b7-xq4xg\" (UID: \"8157c822-ab10-4925-b3c9-6211dff26119\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e14e00-6bfcb896b7-xq4xg"
Oct 30 22:52:19.108409 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:52:19.108350    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8157c822-ab10-4925-b3c9-6211dff26119-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-e14e00-6bfcb896b7-xq4xg\" (UID: \"8157c822-ab10-4925-b3c9-6211dff26119\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e14e00-6bfcb896b7-xq4xg"
Oct 30 22:52:19.108501 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:52:19.108433    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8157c822-ab10-4925-b3c9-6211dff26119-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-e14e00-6bfcb896b7-xq4xg\" (UID: \"8157c822-ab10-4925-b3c9-6211dff26119\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e14e00-6bfcb896b7-xq4xg"
Oct 30 22:52:19.108927 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:52:19.108643    4477 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e14e00-serving-cert: secret "maas-api-e2e-shared-b-w3-e14e00-serving-cert" not found
Oct 30 22:52:19.108927 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:52:19.108720    4477 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8157c822-ab10-4925-b3c9-6211dff26119-maas-api-tls podName:8157c822-ab10-4925-b3c9-6211dff26119 nodeName:}" failed. No retries permitted until 2026-10-30 22:52:19.608693791 +0000 UTC m=+2723.534619989 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8157c822-ab10-4925-b3c9-6211dff26119-maas-api-tls") pod "maas-api-e2e-shared-b-w3-e14e00-6bfcb896b7-xq4xg" (UID: "8157c822-ab10-4925-b3c9-6211dff26119") : secret "maas-api-e2e-shared-b-w3-e14e00-serving-cert" not found
Oct 30 22:52:19.109260 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:52:19.109234    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8157c822-ab10-4925-b3c9-6211dff26119-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-e14e00-6bfcb896b7-xq4xg\" (UID: \"8157c822-ab10-4925-b3c9-6211dff26119\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e14e00-6bfcb896b7-xq4xg"
Oct 30 22:52:19.111608 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:52:19.111584    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8157c822-ab10-4925-b3c9-6211dff26119-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-e14e00-6bfcb896b7-xq4xg\" (UID: \"8157c822-ab10-4925-b3c9-6211dff26119\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e14e00-6bfcb896b7-xq4xg"
Oct 30 22:52:19.132319 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:52:19.132293    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhvkt\" (UniqueName: \"kubernetes.io/projected/8157c822-ab10-4925-b3c9-6211dff26119-kube-api-access-fhvkt\") pod \"maas-api-e2e-shared-b-w3-e14e00-6bfcb896b7-xq4xg\" (UID: \"8157c822-ab10-4925-b3c9-6211dff26119\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e14e00-6bfcb896b7-xq4xg"
Oct 30 22:52:19.614471 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:52:19.614387    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8157c822-ab10-4925-b3c9-6211dff26119-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-e14e00-6bfcb896b7-xq4xg\" (UID: \"8157c822-ab10-4925-b3c9-6211dff26119\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e14e00-6bfcb896b7-xq4xg"
Oct 30 22:52:19.616824 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:52:19.616798    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8157c822-ab10-4925-b3c9-6211dff26119-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-e14e00-6bfcb896b7-xq4xg\" (UID: \"8157c822-ab10-4925-b3c9-6211dff26119\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e14e00-6bfcb896b7-xq4xg"
Oct 30 22:52:19.843835 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:52:19.843793    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e14e00-6bfcb896b7-xq4xg"
Oct 30 22:52:19.894063 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:52:19.882079    4477 patch_prober.go:29] interesting pod/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-6bvgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body=
Oct 30 22:52:19.894063 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:52:19.882147    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-6bvgh" podUID="18a561b5-289d-4401-8377-0be0130d658d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused"
Oct 30 22:52:20.195576 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:52:20.195543    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8157c822_ab10_4925_b3c9_6211dff26119.slice/crio-ba5918f47726bbfb75336f28886891e257248325ae530b3be91ce6b85b948027 WatchSource:0}: Error finding container ba5918f47726bbfb75336f28886891e257248325ae530b3be91ce6b85b948027: Status 404 returned error can't find the container with id ba5918f47726bbfb75336f28886891e257248325ae530b3be91ce6b85b948027
Oct 30 22:52:20.198444 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:52:20.198415    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e14e00-6bfcb896b7-xq4xg"]
Oct 30 22:52:20.872903 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:52:20.872882    4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-6bvgh"
Oct 30 22:52:20.873385 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:52:20.873359    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-6bvgh"
Oct 30 22:52:20.874008 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:52:20.873989    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-6bvgh"
Oct 30 22:52:20.968858 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:52:20.968824    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e14e00-6bfcb896b7-xq4xg" event={"ID":"8157c822-ab10-4925-b3c9-6211dff26119","Type":"ContainerStarted","Data":"c715d01dee2aaf025072e7a92e611d31396fa32e4b4a9a4b55193f32222254e0"}
Oct 30 22:52:20.969034 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:52:20.968872    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e14e00-6bfcb896b7-xq4xg" event={"ID":"8157c822-ab10-4925-b3c9-6211dff26119","Type":"ContainerStarted","Data":"ba5918f47726bbfb75336f28886891e257248325ae530b3be91ce6b85b948027"}
Oct 30 22:52:21.028410 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:52:21.028344    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e14e00-6bfcb896b7-xq4xg" podStartSLOduration=2.698877343 podStartE2EDuration="3.028324082s" totalImagesPullingTime="329.446739ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:52:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:52:21.019753591 +0000 UTC m=+2724.945679812" watchObservedRunningTime="2026-10-30 22:52:21.028324082 +0000 UTC m=+2724.954250289"
Oct 30 22:52:21.975293 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:52:21.975249    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e14e00-6bfcb896b7-xq4xg"
Oct 30 22:52:26.268192 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:52:26.268151    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 22:52:27.987004 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:52:27.986976    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e14e00-6bfcb896b7-xq4xg"
Oct 30 22:52:56.269341 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:52:56.269258    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 22:53:23.786540 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:53:23.786502    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e3a7dd-fc4fb5599-mg28w"]
Oct 30 22:53:23.791851 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:53:23.791305    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e3a7dd-fc4fb5599-mg28w"
Oct 30 22:53:23.803814 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:53:23.803789    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e3a7dd-fc4fb5599-mg28w"]
Oct 30 22:53:23.810824 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:53:23.810797    4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-e3a7dd\""
Oct 30 22:53:23.811163 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:53:23.811136    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-e3a7dd-dockercfg-k2kbq\""
Oct 30 22:53:23.942022 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:53:23.941983    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e3a7dd-6b6b8b4cfd-9gg8g"]
Oct 30 22:53:23.946615 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:53:23.946591    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e3a7dd-6b6b8b4cfd-9gg8g"
Oct 30 22:53:23.947343 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:53:23.947309    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5hjm\" (UniqueName: \"kubernetes.io/projected/c178db77-b489-434c-99e3-c7934518a7f8-kube-api-access-g5hjm\") pod \"payload-pre-processing-e2e-ipp-a-e3a7dd-fc4fb5599-mg28w\" (UID: \"c178db77-b489-434c-99e3-c7934518a7f8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e3a7dd-fc4fb5599-mg28w"
Oct 30 22:53:23.947505 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:53:23.947485    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c178db77-b489-434c-99e3-c7934518a7f8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-e3a7dd-fc4fb5599-mg28w\" (UID: \"c178db77-b489-434c-99e3-c7934518a7f8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e3a7dd-fc4fb5599-mg28w"
Oct 30 22:53:23.969034 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:53:23.969004    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e3a7dd-6b6b8b4cfd-9gg8g"]
Oct 30 22:53:24.049139 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:53:24.049031    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5hjm\" (UniqueName: \"kubernetes.io/projected/c178db77-b489-434c-99e3-c7934518a7f8-kube-api-access-g5hjm\") pod \"payload-pre-processing-e2e-ipp-a-e3a7dd-fc4fb5599-mg28w\" (UID: \"c178db77-b489-434c-99e3-c7934518a7f8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e3a7dd-fc4fb5599-mg28w"
Oct 30 22:53:24.049139 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:53:24.049135    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04451835-ece6-4466-8b50-71aaf24b4828-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-e3a7dd-6b6b8b4cfd-9gg8g\" (UID: \"04451835-ece6-4466-8b50-71aaf24b4828\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e3a7dd-6b6b8b4cfd-9gg8g"
Oct 30 22:53:24.049318 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:53:24.049208    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c178db77-b489-434c-99e3-c7934518a7f8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-e3a7dd-fc4fb5599-mg28w\" (UID: \"c178db77-b489-434c-99e3-c7934518a7f8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e3a7dd-fc4fb5599-mg28w"
Oct 30 22:53:24.049318 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:53:24.049256    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64wqd\" (UniqueName: \"kubernetes.io/projected/04451835-ece6-4466-8b50-71aaf24b4828-kube-api-access-64wqd\") pod \"payload-processing-e2e-ipp-a-e3a7dd-6b6b8b4cfd-9gg8g\" (UID: \"04451835-ece6-4466-8b50-71aaf24b4828\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e3a7dd-6b6b8b4cfd-9gg8g"
Oct 30 22:53:24.049851 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:53:24.049832    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c178db77-b489-434c-99e3-c7934518a7f8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-e3a7dd-fc4fb5599-mg28w\" (UID: \"c178db77-b489-434c-99e3-c7934518a7f8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e3a7dd-fc4fb5599-mg28w"
Oct 30 22:53:24.068234 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:53:24.068201    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5hjm\" (UniqueName: \"kubernetes.io/projected/c178db77-b489-434c-99e3-c7934518a7f8-kube-api-access-g5hjm\") pod \"payload-pre-processing-e2e-ipp-a-e3a7dd-fc4fb5599-mg28w\" (UID: \"c178db77-b489-434c-99e3-c7934518a7f8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e3a7dd-fc4fb5599-mg28w"
Oct 30 22:53:24.118202 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:53:24.118171    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e3a7dd-fc4fb5599-mg28w"
Oct 30 22:53:24.150313 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:53:24.150287    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04451835-ece6-4466-8b50-71aaf24b4828-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-e3a7dd-6b6b8b4cfd-9gg8g\" (UID: \"04451835-ece6-4466-8b50-71aaf24b4828\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e3a7dd-6b6b8b4cfd-9gg8g"
Oct 30 22:53:24.150434 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:53:24.150356    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64wqd\" (UniqueName: \"kubernetes.io/projected/04451835-ece6-4466-8b50-71aaf24b4828-kube-api-access-64wqd\") pod \"payload-processing-e2e-ipp-a-e3a7dd-6b6b8b4cfd-9gg8g\" (UID: \"04451835-ece6-4466-8b50-71aaf24b4828\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e3a7dd-6b6b8b4cfd-9gg8g"
Oct 30 22:53:24.150953 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:53:24.150934    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04451835-ece6-4466-8b50-71aaf24b4828-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-e3a7dd-6b6b8b4cfd-9gg8g\" (UID: \"04451835-ece6-4466-8b50-71aaf24b4828\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e3a7dd-6b6b8b4cfd-9gg8g"
Oct 30 22:53:24.171225 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:53:24.171198    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64wqd\" (UniqueName: \"kubernetes.io/projected/04451835-ece6-4466-8b50-71aaf24b4828-kube-api-access-64wqd\") pod \"payload-processing-e2e-ipp-a-e3a7dd-6b6b8b4cfd-9gg8g\" (UID: \"04451835-ece6-4466-8b50-71aaf24b4828\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e3a7dd-6b6b8b4cfd-9gg8g"
Oct 30 22:53:24.268920 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:53:24.268890    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e3a7dd-6b6b8b4cfd-9gg8g"
Oct 30 22:53:24.421161 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:53:24.421120    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e3a7dd-fc4fb5599-mg28w"]
Oct 30 22:53:24.425394 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:53:24.425366    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc178db77_b489_434c_99e3_c7934518a7f8.slice/crio-7144c86c880b7b4893317cd55f8947a348273559ce58bc708ab6cb7c5411ce95 WatchSource:0}: Error finding container 7144c86c880b7b4893317cd55f8947a348273559ce58bc708ab6cb7c5411ce95: Status 404 returned error can't find the container with id 7144c86c880b7b4893317cd55f8947a348273559ce58bc708ab6cb7c5411ce95
Oct 30 22:53:24.608440 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:53:24.608396    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e3a7dd-6b6b8b4cfd-9gg8g"]
Oct 30 22:53:25.395816 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:53:25.395777    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e3a7dd-6b6b8b4cfd-9gg8g" event={"ID":"04451835-ece6-4466-8b50-71aaf24b4828","Type":"ContainerStarted","Data":"e8faf9a432ce87d7d4b08a94dd75b9ad093df86c81e4aa8c1cf914275b2a3d59"}
Oct 30 22:53:25.397339 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:53:25.397300    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e3a7dd-fc4fb5599-mg28w" event={"ID":"c178db77-b489-434c-99e3-c7934518a7f8","Type":"ContainerStarted","Data":"7144c86c880b7b4893317cd55f8947a348273559ce58bc708ab6cb7c5411ce95"}
Oct 30 22:53:26.269762 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:53:26.269722    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 22:53:26.405975 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:53:26.405938    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e3a7dd-6b6b8b4cfd-9gg8g" event={"ID":"04451835-ece6-4466-8b50-71aaf24b4828","Type":"ContainerStarted","Data":"0b8f3f1cdbda169c2a7c2ea51bdf5003cccf2428fc9b6df4473f8953414805b5"}
Oct 30 22:53:26.406489 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:53:26.406023    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-e3a7dd-6b6b8b4cfd-9gg8g"
Oct 30 22:53:26.407815 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:53:26.407783    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e3a7dd-fc4fb5599-mg28w" event={"ID":"c178db77-b489-434c-99e3-c7934518a7f8","Type":"ContainerStarted","Data":"43ef41e7bf29dfd1a3369830f718bbb11eedecd6381a8f283a099f8f5ab4ce3a"}
Oct 30 22:53:26.407975 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:53:26.407869    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e3a7dd-fc4fb5599-mg28w"
Oct 30 22:53:26.446164 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:53:26.446123    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-e3a7dd-6b6b8b4cfd-9gg8g" podStartSLOduration=2.232371928 podStartE2EDuration="3.446111069s" totalImagesPullingTime="1.213739141s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:53:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:53:26.445378651 +0000 UTC m=+2790.371304858" watchObservedRunningTime="2026-10-30 22:53:26.446111069 +0000 UTC m=+2790.372037275"
Oct 30 22:53:26.490004 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:53:26.489954    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e3a7dd-fc4fb5599-mg28w" podStartSLOduration=2.098388747 podStartE2EDuration="3.489938131s" totalImagesPullingTime="1.391549384s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:53:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:53:26.48899254 +0000 UTC m=+2790.414918772" watchObservedRunningTime="2026-10-30 22:53:26.489938131 +0000 UTC m=+2790.415864339"
Oct 30 22:53:31.462850 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:53:31.462818    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a60e80-6fdbc5f7f9-24lzn"]
Oct 30 22:53:31.467104 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:53:31.467085    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a60e80-6fdbc5f7f9-24lzn"
Oct 30 22:53:31.484680 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:53:31.484654    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-a60e80-serving-cert\""
Oct 30 22:53:31.496104 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:53:31.496078    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a60e80-6fdbc5f7f9-24lzn"]
Oct 30 22:53:31.536329 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:53:31.536293    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d034fb1b-68f3-437a-ba54-f4537069c282-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-a60e80-6fdbc5f7f9-24lzn\" (UID: \"d034fb1b-68f3-437a-ba54-f4537069c282\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a60e80-6fdbc5f7f9-24lzn"
Oct 30 22:53:31.536521 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:53:31.536383    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsvxp\" (UniqueName: \"kubernetes.io/projected/d034fb1b-68f3-437a-ba54-f4537069c282-kube-api-access-jsvxp\") pod \"maas-api-e2e-ipp-b-a60e80-6fdbc5f7f9-24lzn\" (UID: \"d034fb1b-68f3-437a-ba54-f4537069c282\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a60e80-6fdbc5f7f9-24lzn"
Oct 30 22:53:31.536591 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:53:31.536554    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d034fb1b-68f3-437a-ba54-f4537069c282-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a60e80-6fdbc5f7f9-24lzn\" (UID: \"d034fb1b-68f3-437a-ba54-f4537069c282\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a60e80-6fdbc5f7f9-24lzn"
Oct 30 22:53:31.536591 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:53:31.536587    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d034fb1b-68f3-437a-ba54-f4537069c282-metrics-certs\") pod \"maas-api-e2e-ipp-b-a60e80-6fdbc5f7f9-24lzn\" (UID: \"d034fb1b-68f3-437a-ba54-f4537069c282\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a60e80-6fdbc5f7f9-24lzn"
Oct 30 22:53:31.637978 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:53:31.637950    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsvxp\" (UniqueName: \"kubernetes.io/projected/d034fb1b-68f3-437a-ba54-f4537069c282-kube-api-access-jsvxp\") pod \"maas-api-e2e-ipp-b-a60e80-6fdbc5f7f9-24lzn\" (UID: \"d034fb1b-68f3-437a-ba54-f4537069c282\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a60e80-6fdbc5f7f9-24lzn"
Oct 30 22:53:31.638165 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:53:31.638069    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d034fb1b-68f3-437a-ba54-f4537069c282-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a60e80-6fdbc5f7f9-24lzn\" (UID: \"d034fb1b-68f3-437a-ba54-f4537069c282\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a60e80-6fdbc5f7f9-24lzn"
Oct 30 22:53:31.638165 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:53:31.638099    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d034fb1b-68f3-437a-ba54-f4537069c282-metrics-certs\") pod \"maas-api-e2e-ipp-b-a60e80-6fdbc5f7f9-24lzn\" (UID: \"d034fb1b-68f3-437a-ba54-f4537069c282\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a60e80-6fdbc5f7f9-24lzn"
Oct 30 22:53:31.638262 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:53:31.638202    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d034fb1b-68f3-437a-ba54-f4537069c282-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-a60e80-6fdbc5f7f9-24lzn\" (UID: \"d034fb1b-68f3-437a-ba54-f4537069c282\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a60e80-6fdbc5f7f9-24lzn"
Oct 30 22:53:31.638919 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:53:31.638892    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d034fb1b-68f3-437a-ba54-f4537069c282-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-a60e80-6fdbc5f7f9-24lzn\" (UID: \"d034fb1b-68f3-437a-ba54-f4537069c282\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a60e80-6fdbc5f7f9-24lzn"
Oct 30 22:53:31.640626 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:53:31.640592    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d034fb1b-68f3-437a-ba54-f4537069c282-metrics-certs\") pod \"maas-api-e2e-ipp-b-a60e80-6fdbc5f7f9-24lzn\" (UID: \"d034fb1b-68f3-437a-ba54-f4537069c282\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a60e80-6fdbc5f7f9-24lzn"
Oct 30 22:53:31.640882 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:53:31.640860    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d034fb1b-68f3-437a-ba54-f4537069c282-maas-api-tls\") pod \"maas-api-e2e-ipp-b-a60e80-6fdbc5f7f9-24lzn\" (UID: \"d034fb1b-68f3-437a-ba54-f4537069c282\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a60e80-6fdbc5f7f9-24lzn"
Oct 30 22:53:31.657152 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:53:31.657134    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsvxp\" (UniqueName: \"kubernetes.io/projected/d034fb1b-68f3-437a-ba54-f4537069c282-kube-api-access-jsvxp\") pod \"maas-api-e2e-ipp-b-a60e80-6fdbc5f7f9-24lzn\" (UID: \"d034fb1b-68f3-437a-ba54-f4537069c282\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a60e80-6fdbc5f7f9-24lzn"
Oct 30 22:53:31.781377 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:53:31.781298    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a60e80-6fdbc5f7f9-24lzn"
Oct 30 22:53:32.093114 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:53:32.093022    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a60e80-6fdbc5f7f9-24lzn"]
Oct 30 22:53:32.096580 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:53:32.096539    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd034fb1b_68f3_437a_ba54_f4537069c282.slice/crio-6677bcdfb3dc7eac69131c418e73877de2a4ba0715f8bb84f678b17d67e37c3c WatchSource:0}: Error finding container 6677bcdfb3dc7eac69131c418e73877de2a4ba0715f8bb84f678b17d67e37c3c: Status 404 returned error can't find the container with id 6677bcdfb3dc7eac69131c418e73877de2a4ba0715f8bb84f678b17d67e37c3c
Oct 30 22:53:32.179004 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:53:32.178968    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6bz92"]
Oct 30 22:53:32.448817 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:53:32.448782    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a60e80-6fdbc5f7f9-24lzn" event={"ID":"d034fb1b-68f3-437a-ba54-f4537069c282","Type":"ContainerStarted","Data":"6677bcdfb3dc7eac69131c418e73877de2a4ba0715f8bb84f678b17d67e37c3c"}
Oct 30 22:53:33.461455 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:53:33.461406    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a60e80-6fdbc5f7f9-24lzn" event={"ID":"d034fb1b-68f3-437a-ba54-f4537069c282","Type":"ContainerStarted","Data":"4913e6b3dfc0b44fa71df9de18b9027df075d7d7f6a1e72ac7cc806cbb80d7c9"}
Oct 30 22:53:33.462442 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:53:33.462419    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a60e80-6fdbc5f7f9-24lzn"
Oct 30 22:53:33.517523 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:53:33.517453    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a60e80-6fdbc5f7f9-24lzn" podStartSLOduration=2.10279527 podStartE2EDuration="2.517436417s" totalImagesPullingTime="414.641147ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:53:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:53:33.51642766 +0000 UTC m=+2797.442353868" watchObservedRunningTime="2026-10-30 22:53:33.517436417 +0000 UTC m=+2797.443362624"
Oct 30 22:53:37.416738 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:53:37.416706    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-e3a7dd-6b6b8b4cfd-9gg8g"
Oct 30 22:53:37.417167 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:53:37.417025    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e3a7dd-fc4fb5599-mg28w"
Oct 30 22:53:39.474085 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:53:39.474055    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a60e80-6fdbc5f7f9-24lzn"
Oct 30 22:53:56.268186 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:53:56.268142    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 22:54:26.270494 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:54:26.270452    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 22:54:56.267298 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:54:56.267261    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 22:55:21.575057 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:55:21.575007    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6bz92"]
Oct 30 22:55:26.270961 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:55:26.270915    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 22:55:54.977129 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:55:54.977033    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-bmccc"]
Oct 30 22:55:54.980821 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:55:54.980800    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-bmccc"
Oct 30 22:55:54.995244 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:55:54.995223    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-3cd77f-openshift-default-dockercfg-vlt6j\""
Oct 30 22:55:55.011569 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:55:55.011545    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-bmccc"]
Oct 30 22:55:55.068070 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:55:55.068008    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30fed99f-b5e6-4c12-a1f4-8d84ed15971e-istio-data\") pod \"e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-bmccc\" (UID: \"30fed99f-b5e6-4c12-a1f4-8d84ed15971e\") " pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-bmccc"
Oct 30 22:55:55.068222 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:55:55.068151    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30fed99f-b5e6-4c12-a1f4-8d84ed15971e-istio-token\") pod \"e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-bmccc\" (UID: \"30fed99f-b5e6-4c12-a1f4-8d84ed15971e\") " pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-bmccc"
Oct 30 22:55:55.068280 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:55:55.068233    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30fed99f-b5e6-4c12-a1f4-8d84ed15971e-workload-socket\") pod \"e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-bmccc\" (UID: \"30fed99f-b5e6-4c12-a1f4-8d84ed15971e\") " pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-bmccc"
Oct 30 22:55:55.068400 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:55:55.068384    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30fed99f-b5e6-4c12-a1f4-8d84ed15971e-credential-socket\") pod \"e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-bmccc\" (UID: \"30fed99f-b5e6-4c12-a1f4-8d84ed15971e\") " pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-bmccc"
Oct 30 22:55:55.068446 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:55:55.068428    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30fed99f-b5e6-4c12-a1f4-8d84ed15971e-istio-podinfo\") pod \"e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-bmccc\" (UID: \"30fed99f-b5e6-4c12-a1f4-8d84ed15971e\") " pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-bmccc"
Oct 30 22:55:55.068513 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:55:55.068498    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30fed99f-b5e6-4c12-a1f4-8d84ed15971e-istiod-ca-cert\") pod \"e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-bmccc\" (UID: \"30fed99f-b5e6-4c12-a1f4-8d84ed15971e\") " pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-bmccc"
Oct 30 22:55:55.068635 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:55:55.068619    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6dhb\" (UniqueName: \"kubernetes.io/projected/30fed99f-b5e6-4c12-a1f4-8d84ed15971e-kube-api-access-j6dhb\") pod \"e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-bmccc\" (UID: \"30fed99f-b5e6-4c12-a1f4-8d84ed15971e\") " pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-bmccc"
Oct 30 22:55:55.068695 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:55:55.068680    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30fed99f-b5e6-4c12-a1f4-8d84ed15971e-istio-envoy\") pod \"e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-bmccc\" (UID: \"30fed99f-b5e6-4c12-a1f4-8d84ed15971e\") " pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-bmccc"
Oct 30 22:55:55.068737 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:55:55.068706    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30fed99f-b5e6-4c12-a1f4-8d84ed15971e-workload-certs\") pod \"e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-bmccc\" (UID: \"30fed99f-b5e6-4c12-a1f4-8d84ed15971e\") " pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-bmccc"
Oct 30 22:55:55.169913 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:55:55.169877    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30fed99f-b5e6-4c12-a1f4-8d84ed15971e-istio-podinfo\") pod \"e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-bmccc\" (UID: \"30fed99f-b5e6-4c12-a1f4-8d84ed15971e\") " pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-bmccc"
Oct 30 22:55:55.170121 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:55:55.169922    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30fed99f-b5e6-4c12-a1f4-8d84ed15971e-istiod-ca-cert\") pod \"e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-bmccc\" (UID: \"30fed99f-b5e6-4c12-a1f4-8d84ed15971e\") " pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-bmccc"
Oct 30 22:55:55.170121 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:55:55.169983    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6dhb\" (UniqueName: \"kubernetes.io/projected/30fed99f-b5e6-4c12-a1f4-8d84ed15971e-kube-api-access-j6dhb\") pod \"e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-bmccc\" (UID: \"30fed99f-b5e6-4c12-a1f4-8d84ed15971e\") " pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-bmccc"
Oct 30 22:55:55.170121 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:55:55.170014    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30fed99f-b5e6-4c12-a1f4-8d84ed15971e-istio-envoy\") pod \"e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-bmccc\" (UID: \"30fed99f-b5e6-4c12-a1f4-8d84ed15971e\") " pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-bmccc"
Oct 30 22:55:55.170121 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:55:55.170034    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30fed99f-b5e6-4c12-a1f4-8d84ed15971e-workload-certs\") pod \"e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-bmccc\" (UID: \"30fed99f-b5e6-4c12-a1f4-8d84ed15971e\") " pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-bmccc"
Oct 30 22:55:55.170121 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:55:55.170100    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30fed99f-b5e6-4c12-a1f4-8d84ed15971e-istio-data\") pod \"e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-bmccc\" (UID: \"30fed99f-b5e6-4c12-a1f4-8d84ed15971e\") " pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-bmccc"
Oct 30 22:55:55.170407 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:55:55.170141    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30fed99f-b5e6-4c12-a1f4-8d84ed15971e-istio-token\") pod \"e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-bmccc\" (UID: \"30fed99f-b5e6-4c12-a1f4-8d84ed15971e\") " pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-bmccc"
Oct 30 22:55:55.170407 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:55:55.170207    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30fed99f-b5e6-4c12-a1f4-8d84ed15971e-workload-socket\") pod \"e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-bmccc\" (UID: \"30fed99f-b5e6-4c12-a1f4-8d84ed15971e\") " pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-bmccc"
Oct 30 22:55:55.170407 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:55:55.170288    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30fed99f-b5e6-4c12-a1f4-8d84ed15971e-credential-socket\") pod \"e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-bmccc\" (UID: \"30fed99f-b5e6-4c12-a1f4-8d84ed15971e\") " pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-bmccc"
Oct 30 22:55:55.170583 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:55:55.170566    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30fed99f-b5e6-4c12-a1f4-8d84ed15971e-istio-data\") pod \"e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-bmccc\" (UID: \"30fed99f-b5e6-4c12-a1f4-8d84ed15971e\") " pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-bmccc"
Oct 30 22:55:55.170686 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:55:55.170659    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30fed99f-b5e6-4c12-a1f4-8d84ed15971e-credential-socket\") pod \"e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-bmccc\" (UID: \"30fed99f-b5e6-4c12-a1f4-8d84ed15971e\") " pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-bmccc"
Oct 30 22:55:55.170792 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:55:55.170764    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30fed99f-b5e6-4c12-a1f4-8d84ed15971e-workload-certs\") pod \"e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-bmccc\" (UID: \"30fed99f-b5e6-4c12-a1f4-8d84ed15971e\") " pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-bmccc"
Oct 30 22:55:55.170855 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:55:55.170812    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30fed99f-b5e6-4c12-a1f4-8d84ed15971e-workload-socket\") pod \"e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-bmccc\" (UID: \"30fed99f-b5e6-4c12-a1f4-8d84ed15971e\") " pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-bmccc"
Oct 30 22:55:55.170918 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:55:55.170867    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30fed99f-b5e6-4c12-a1f4-8d84ed15971e-istiod-ca-cert\") pod \"e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-bmccc\" (UID: \"30fed99f-b5e6-4c12-a1f4-8d84ed15971e\") " pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-bmccc"
Oct 30 22:55:55.172298 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:55:55.172270    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30fed99f-b5e6-4c12-a1f4-8d84ed15971e-istio-envoy\") pod \"e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-bmccc\" (UID: \"30fed99f-b5e6-4c12-a1f4-8d84ed15971e\") " pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-bmccc"
Oct 30 22:55:55.172444 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:55:55.172428    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30fed99f-b5e6-4c12-a1f4-8d84ed15971e-istio-podinfo\") pod \"e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-bmccc\" (UID: \"30fed99f-b5e6-4c12-a1f4-8d84ed15971e\") " pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-bmccc"
Oct 30 22:55:55.189361 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:55:55.189333    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30fed99f-b5e6-4c12-a1f4-8d84ed15971e-istio-token\") pod \"e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-bmccc\" (UID: \"30fed99f-b5e6-4c12-a1f4-8d84ed15971e\") " pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-bmccc"
Oct 30 22:55:55.189466 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:55:55.189378    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6dhb\" (UniqueName: \"kubernetes.io/projected/30fed99f-b5e6-4c12-a1f4-8d84ed15971e-kube-api-access-j6dhb\") pod \"e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-bmccc\" (UID: \"30fed99f-b5e6-4c12-a1f4-8d84ed15971e\") " pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-bmccc"
Oct 30 22:55:55.294655 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:55:55.294602    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-bmccc"
Oct 30 22:55:55.615198 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:55:55.614995    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-bmccc"]
Oct 30 22:55:55.618363 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:55:55.618335    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30fed99f_b5e6_4c12_a1f4_8d84ed15971e.slice/crio-6b9c41ce39a073f91a5e82778075dc3f4e32af70ef63d09e6a955a3cbf207335 WatchSource:0}: Error finding container 6b9c41ce39a073f91a5e82778075dc3f4e32af70ef63d09e6a955a3cbf207335: Status 404 returned error can't find the container with id 6b9c41ce39a073f91a5e82778075dc3f4e32af70ef63d09e6a955a3cbf207335
Oct 30 22:55:55.621026 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:55:55.620988    4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:55:55.621148 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:55:55.621089    4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:55:55.621148 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:55:55.621139    4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 22:55:56.267978 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:55:56.267944    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 22:55:56.501229 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:55:56.501183    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-bmccc" event={"ID":"30fed99f-b5e6-4c12-a1f4-8d84ed15971e","Type":"ContainerStarted","Data":"0781d508d5529cb9d8676b66eafa6284ea19d8b76451ac40ce1aac9910076060"}
Oct 30 22:55:56.501397 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:55:56.501255    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-bmccc" event={"ID":"30fed99f-b5e6-4c12-a1f4-8d84ed15971e","Type":"ContainerStarted","Data":"6b9c41ce39a073f91a5e82778075dc3f4e32af70ef63d09e6a955a3cbf207335"}
Oct 30 22:55:56.558433 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:55:56.558341    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-bmccc" podStartSLOduration=2.558327683 podStartE2EDuration="2.558327683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:55:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:55:56.55532 +0000 UTC m=+2940.481246231" watchObservedRunningTime="2026-10-30 22:55:56.558327683 +0000 UTC m=+2940.484253888"
Oct 30 22:55:57.295465 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:55:57.295424    4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-bmccc"
Oct 30 22:55:57.296872 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:55:57.296829    4477 patch_prober.go:29] interesting pod/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-bmccc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body=
Oct 30 22:55:57.297010 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:55:57.296890    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-bmccc" podUID="30fed99f-b5e6-4c12-a1f4-8d84ed15971e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused"
Oct 30 22:55:58.296066 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:55:58.295087    4477 patch_prober.go:29] interesting pod/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-bmccc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body=
Oct 30 22:55:58.296066 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:55:58.295145    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-bmccc" podUID="30fed99f-b5e6-4c12-a1f4-8d84ed15971e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused"
Oct 30 22:55:59.295827 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:55:59.295793    4477 patch_prober.go:29] interesting pod/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-bmccc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body=
Oct 30 22:55:59.296033 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:55:59.295850    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-bmccc" podUID="30fed99f-b5e6-4c12-a1f4-8d84ed15971e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused"
Oct 30 22:56:00.295615 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:56:00.295572    4477 patch_prober.go:29] interesting pod/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-bmccc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body=
Oct 30 22:56:00.296162 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:56:00.295639    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-bmccc" podUID="30fed99f-b5e6-4c12-a1f4-8d84ed15971e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused"
Oct 30 22:56:01.295503 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:56:01.295460    4477 patch_prober.go:29] interesting pod/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-bmccc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body=
Oct 30 22:56:01.295719 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:56:01.295513    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-bmccc" podUID="30fed99f-b5e6-4c12-a1f4-8d84ed15971e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused"
Oct 30 22:56:02.299837 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:56:02.299807    4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-bmccc"
Oct 30 22:56:02.300332 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:56:02.300098    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-bmccc"
Oct 30 22:56:02.301175 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:56:02.301157    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-bmccc"
Oct 30 22:56:04.938854 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:56:04.938821    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-5983ea/auth-test-model-5983ea-kserve-d4df66dd5-96zv4"]
Oct 30 22:56:17.061176 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:56:17.061143    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-b03b33/auth-test-model-b03b33-kserve-d99bb5cb9-fzz5k"]
Oct 30 22:56:19.917059 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:56:19.916992    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-80ecb3/sub-test-model-80ecb3-kserve-9b4b44659-vnh9d"]
Oct 30 22:56:19.967589 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:56:19.967519    4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w0-5983ea/auth-test-model-5983ea-kserve-d4df66dd5-96zv4" podUID="1e727c25-f834-4b00-aec6-9f3c7cb0b97b" containerName="main" containerID="cri-o://82f57cc8dcc27306103ee99ebc7f0cf34d7a1437428cf0405dc8eb30ac38c2f0" gracePeriod=45
Oct 30 22:56:20.308388 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:56:20.308356    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-5983ea/auth-test-model-5983ea-kserve-d4df66dd5-96zv4"
Oct 30 22:56:20.354211 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:56:20.354178    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e727c25-f834-4b00-aec6-9f3c7cb0b97b-tls-certs\" (UniqueName: \"kubernetes.io/secret/1e727c25-f834-4b00-aec6-9f3c7cb0b97b-tls-certs\") pod \"1e727c25-f834-4b00-aec6-9f3c7cb0b97b\" (UID: \"1e727c25-f834-4b00-aec6-9f3c7cb0b97b\") "
Oct 30 22:56:20.354390 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:56:20.354353    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e727c25-f834-4b00-aec6-9f3c7cb0b97b-kube-api-access-hbjmg\" (UniqueName: \"kubernetes.io/projected/1e727c25-f834-4b00-aec6-9f3c7cb0b97b-kube-api-access-hbjmg\") pod \"1e727c25-f834-4b00-aec6-9f3c7cb0b97b\" (UID: \"1e727c25-f834-4b00-aec6-9f3c7cb0b97b\") "
Oct 30 22:56:20.354461 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:56:20.354439    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e727c25-f834-4b00-aec6-9f3c7cb0b97b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e727c25-f834-4b00-aec6-9f3c7cb0b97b-tmp-dir\") pod \"1e727c25-f834-4b00-aec6-9f3c7cb0b97b\" (UID: \"1e727c25-f834-4b00-aec6-9f3c7cb0b97b\") "
Oct 30 22:56:20.354531 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:56:20.354507    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e727c25-f834-4b00-aec6-9f3c7cb0b97b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e727c25-f834-4b00-aec6-9f3c7cb0b97b-model-cache\") pod \"1e727c25-f834-4b00-aec6-9f3c7cb0b97b\" (UID: \"1e727c25-f834-4b00-aec6-9f3c7cb0b97b\") "
Oct 30 22:56:20.354634 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:56:20.354540    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e727c25-f834-4b00-aec6-9f3c7cb0b97b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e727c25-f834-4b00-aec6-9f3c7cb0b97b-dshm\") pod \"1e727c25-f834-4b00-aec6-9f3c7cb0b97b\" (UID: \"1e727c25-f834-4b00-aec6-9f3c7cb0b97b\") "
Oct 30 22:56:20.354738 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:56:20.354654    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e727c25-f834-4b00-aec6-9f3c7cb0b97b-home\" (UniqueName: \"kubernetes.io/empty-dir/1e727c25-f834-4b00-aec6-9f3c7cb0b97b-home\") pod \"1e727c25-f834-4b00-aec6-9f3c7cb0b97b\" (UID: \"1e727c25-f834-4b00-aec6-9f3c7cb0b97b\") "
Oct 30 22:56:20.355415 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:56:20.355374    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e727c25-f834-4b00-aec6-9f3c7cb0b97b-home" pod "1e727c25-f834-4b00-aec6-9f3c7cb0b97b" (UID: "1e727c25-f834-4b00-aec6-9f3c7cb0b97b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:56:20.355577 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:56:20.355424    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e727c25-f834-4b00-aec6-9f3c7cb0b97b-tmp-dir" pod "1e727c25-f834-4b00-aec6-9f3c7cb0b97b" (UID: "1e727c25-f834-4b00-aec6-9f3c7cb0b97b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:56:20.356315 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:56:20.355787    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e727c25-f834-4b00-aec6-9f3c7cb0b97b-model-cache" pod "1e727c25-f834-4b00-aec6-9f3c7cb0b97b" (UID: "1e727c25-f834-4b00-aec6-9f3c7cb0b97b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:56:20.358298 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:56:20.357876    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e727c25-f834-4b00-aec6-9f3c7cb0b97b-dshm" pod "1e727c25-f834-4b00-aec6-9f3c7cb0b97b" (UID: "1e727c25-f834-4b00-aec6-9f3c7cb0b97b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:56:20.358298 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:56:20.357947    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e727c25-f834-4b00-aec6-9f3c7cb0b97b-kube-api-access-hbjmg" pod "1e727c25-f834-4b00-aec6-9f3c7cb0b97b" (UID: "1e727c25-f834-4b00-aec6-9f3c7cb0b97b"). InnerVolumeSpecName "kube-api-access-hbjmg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:56:20.358298 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:56:20.358082    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e727c25-f834-4b00-aec6-9f3c7cb0b97b-tls-certs" pod "1e727c25-f834-4b00-aec6-9f3c7cb0b97b" (UID: "1e727c25-f834-4b00-aec6-9f3c7cb0b97b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:56:20.456622 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:56:20.456546    4477 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e727c25-f834-4b00-aec6-9f3c7cb0b97b-tmp-dir\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:56:20.456622 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:56:20.456574    4477 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e727c25-f834-4b00-aec6-9f3c7cb0b97b-model-cache\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:56:20.456622 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:56:20.456583    4477 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e727c25-f834-4b00-aec6-9f3c7cb0b97b-dshm\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:56:20.456622 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:56:20.456590    4477 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e727c25-f834-4b00-aec6-9f3c7cb0b97b-home\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:56:20.456622 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:56:20.456598    4477 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e727c25-f834-4b00-aec6-9f3c7cb0b97b-tls-certs\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:56:20.456622 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:56:20.456606    4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbjmg\" (UniqueName: \"kubernetes.io/projected/1e727c25-f834-4b00-aec6-9f3c7cb0b97b-kube-api-access-hbjmg\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:56:20.709685 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:56:20.709614    4477 generic.go:356] "Generic (PLEG): container finished" podID="1e727c25-f834-4b00-aec6-9f3c7cb0b97b" containerID="82f57cc8dcc27306103ee99ebc7f0cf34d7a1437428cf0405dc8eb30ac38c2f0" exitCode=0
Oct 30 22:56:20.709685 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:56:20.709655    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-5983ea/auth-test-model-5983ea-kserve-d4df66dd5-96zv4"
Oct 30 22:56:20.709892 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:56:20.709691    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-5983ea/auth-test-model-5983ea-kserve-d4df66dd5-96zv4" event={"ID":"1e727c25-f834-4b00-aec6-9f3c7cb0b97b","Type":"ContainerDied","Data":"82f57cc8dcc27306103ee99ebc7f0cf34d7a1437428cf0405dc8eb30ac38c2f0"}
Oct 30 22:56:20.709892 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:56:20.709736    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-5983ea/auth-test-model-5983ea-kserve-d4df66dd5-96zv4" event={"ID":"1e727c25-f834-4b00-aec6-9f3c7cb0b97b","Type":"ContainerDied","Data":"2cee43baca258c605911f9130477fdb256209bdc28b5120e18700d098888660e"}
Oct 30 22:56:20.709892 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:56:20.709756    4477 scope.go:122] "RemoveContainer" containerID="82f57cc8dcc27306103ee99ebc7f0cf34d7a1437428cf0405dc8eb30ac38c2f0"
Oct 30 22:56:20.720326 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:56:20.720308    4477 scope.go:122] "RemoveContainer" containerID="82f57cc8dcc27306103ee99ebc7f0cf34d7a1437428cf0405dc8eb30ac38c2f0"
Oct 30 22:56:20.720578 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:56:20.720558    4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82f57cc8dcc27306103ee99ebc7f0cf34d7a1437428cf0405dc8eb30ac38c2f0\": container with ID starting with 82f57cc8dcc27306103ee99ebc7f0cf34d7a1437428cf0405dc8eb30ac38c2f0 not found: ID does not exist" containerID="82f57cc8dcc27306103ee99ebc7f0cf34d7a1437428cf0405dc8eb30ac38c2f0"
Oct 30 22:56:20.720638 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:56:20.720589    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82f57cc8dcc27306103ee99ebc7f0cf34d7a1437428cf0405dc8eb30ac38c2f0"} err="failed to get container status \"82f57cc8dcc27306103ee99ebc7f0cf34d7a1437428cf0405dc8eb30ac38c2f0\": rpc error: code = NotFound desc = could not find container \"82f57cc8dcc27306103ee99ebc7f0cf34d7a1437428cf0405dc8eb30ac38c2f0\": container with ID starting with 82f57cc8dcc27306103ee99ebc7f0cf34d7a1437428cf0405dc8eb30ac38c2f0 not found: ID does not exist"
Oct 30 22:56:20.761491 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:56:20.761467    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-5983ea/auth-test-model-5983ea-kserve-d4df66dd5-96zv4"]
Oct 30 22:56:20.772641 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:56:20.772618    4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w0-5983ea/auth-test-model-5983ea-kserve-d4df66dd5-96zv4"]
Oct 30 22:56:22.194847 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:56:22.194818    4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e727c25-f834-4b00-aec6-9f3c7cb0b97b" path="/var/lib/kubelet/pods/1e727c25-f834-4b00-aec6-9f3c7cb0b97b/volumes"
Oct 30 22:56:25.588390 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:56:25.588354    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6bz92"]
Oct 30 22:56:26.267369 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:56:26.267334    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 22:56:32.123591 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:56:32.123533    4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-b03b33/auth-test-model-b03b33-kserve-d99bb5cb9-fzz5k" podUID="ae4a262e-38d8-493c-95f8-1bf666eff305" containerName="main" containerID="cri-o://cb6e55d9badb01455dbb30f16c9da20ea07c3db38284d38cff68f4feeffae420" gracePeriod=45
Oct 30 22:56:32.388166 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:56:32.388109    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-b03b33/auth-test-model-b03b33-kserve-d99bb5cb9-fzz5k"
Oct 30 22:56:32.479576 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:56:32.479534    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae4a262e-38d8-493c-95f8-1bf666eff305-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae4a262e-38d8-493c-95f8-1bf666eff305-model-cache\") pod \"ae4a262e-38d8-493c-95f8-1bf666eff305\" (UID: \"ae4a262e-38d8-493c-95f8-1bf666eff305\") "
Oct 30 22:56:32.479744 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:56:32.479585    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae4a262e-38d8-493c-95f8-1bf666eff305-home\" (UniqueName: \"kubernetes.io/empty-dir/ae4a262e-38d8-493c-95f8-1bf666eff305-home\") pod \"ae4a262e-38d8-493c-95f8-1bf666eff305\" (UID: \"ae4a262e-38d8-493c-95f8-1bf666eff305\") "
Oct 30 22:56:32.479744 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:56:32.479613    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae4a262e-38d8-493c-95f8-1bf666eff305-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae4a262e-38d8-493c-95f8-1bf666eff305-tmp-dir\") pod \"ae4a262e-38d8-493c-95f8-1bf666eff305\" (UID: \"ae4a262e-38d8-493c-95f8-1bf666eff305\") "
Oct 30 22:56:32.479744 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:56:32.479643    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae4a262e-38d8-493c-95f8-1bf666eff305-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae4a262e-38d8-493c-95f8-1bf666eff305-dshm\") pod \"ae4a262e-38d8-493c-95f8-1bf666eff305\" (UID: \"ae4a262e-38d8-493c-95f8-1bf666eff305\") "
Oct 30 22:56:32.479744 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:56:32.479712    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae4a262e-38d8-493c-95f8-1bf666eff305-kube-api-access-l85f6\" (UniqueName: \"kubernetes.io/projected/ae4a262e-38d8-493c-95f8-1bf666eff305-kube-api-access-l85f6\") pod \"ae4a262e-38d8-493c-95f8-1bf666eff305\" (UID: \"ae4a262e-38d8-493c-95f8-1bf666eff305\") "
Oct 30 22:56:32.479982 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:56:32.479761    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae4a262e-38d8-493c-95f8-1bf666eff305-tls-certs\" (UniqueName: \"kubernetes.io/secret/ae4a262e-38d8-493c-95f8-1bf666eff305-tls-certs\") pod \"ae4a262e-38d8-493c-95f8-1bf666eff305\" (UID: \"ae4a262e-38d8-493c-95f8-1bf666eff305\") "
Oct 30 22:56:32.479982 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:56:32.479845    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae4a262e-38d8-493c-95f8-1bf666eff305-model-cache" pod "ae4a262e-38d8-493c-95f8-1bf666eff305" (UID: "ae4a262e-38d8-493c-95f8-1bf666eff305"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:56:32.479982 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:56:32.479857    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae4a262e-38d8-493c-95f8-1bf666eff305-home" pod "ae4a262e-38d8-493c-95f8-1bf666eff305" (UID: "ae4a262e-38d8-493c-95f8-1bf666eff305"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:56:32.479982 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:56:32.479969    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae4a262e-38d8-493c-95f8-1bf666eff305-tmp-dir" pod "ae4a262e-38d8-493c-95f8-1bf666eff305" (UID: "ae4a262e-38d8-493c-95f8-1bf666eff305"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:56:32.480411 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:56:32.480347    4477 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae4a262e-38d8-493c-95f8-1bf666eff305-tmp-dir\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:56:32.480411 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:56:32.480372    4477 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae4a262e-38d8-493c-95f8-1bf666eff305-model-cache\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:56:32.480411 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:56:32.480385    4477 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae4a262e-38d8-493c-95f8-1bf666eff305-home\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:56:32.482674 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:56:32.482646    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae4a262e-38d8-493c-95f8-1bf666eff305-kube-api-access-l85f6" pod "ae4a262e-38d8-493c-95f8-1bf666eff305" (UID: "ae4a262e-38d8-493c-95f8-1bf666eff305"). InnerVolumeSpecName "kube-api-access-l85f6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:56:32.482779 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:56:32.482729    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae4a262e-38d8-493c-95f8-1bf666eff305-tls-certs" pod "ae4a262e-38d8-493c-95f8-1bf666eff305" (UID: "ae4a262e-38d8-493c-95f8-1bf666eff305"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:56:32.486724 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:56:32.486701    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae4a262e-38d8-493c-95f8-1bf666eff305-dshm" pod "ae4a262e-38d8-493c-95f8-1bf666eff305" (UID: "ae4a262e-38d8-493c-95f8-1bf666eff305"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:56:32.582183 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:56:32.582152    4477 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae4a262e-38d8-493c-95f8-1bf666eff305-dshm\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:56:32.582183 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:56:32.582180    4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l85f6\" (UniqueName: \"kubernetes.io/projected/ae4a262e-38d8-493c-95f8-1bf666eff305-kube-api-access-l85f6\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:56:32.582183 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:56:32.582190    4477 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae4a262e-38d8-493c-95f8-1bf666eff305-tls-certs\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:56:32.804197 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:56:32.804113    4477 generic.go:356] "Generic (PLEG): container finished" podID="ae4a262e-38d8-493c-95f8-1bf666eff305" containerID="cb6e55d9badb01455dbb30f16c9da20ea07c3db38284d38cff68f4feeffae420" exitCode=0
Oct 30 22:56:32.804197 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:56:32.804162    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-b03b33/auth-test-model-b03b33-kserve-d99bb5cb9-fzz5k"
Oct 30 22:56:32.804412 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:56:32.804197    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-b03b33/auth-test-model-b03b33-kserve-d99bb5cb9-fzz5k" event={"ID":"ae4a262e-38d8-493c-95f8-1bf666eff305","Type":"ContainerDied","Data":"cb6e55d9badb01455dbb30f16c9da20ea07c3db38284d38cff68f4feeffae420"}
Oct 30 22:56:32.804412 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:56:32.804243    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-b03b33/auth-test-model-b03b33-kserve-d99bb5cb9-fzz5k" event={"ID":"ae4a262e-38d8-493c-95f8-1bf666eff305","Type":"ContainerDied","Data":"7f6a4df5b9f77932099b3b166b3657c2b38d727b3ebf524507a9042cf71e1f02"}
Oct 30 22:56:32.804412 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:56:32.804261    4477 scope.go:122] "RemoveContainer" containerID="cb6e55d9badb01455dbb30f16c9da20ea07c3db38284d38cff68f4feeffae420"
Oct 30 22:56:32.814590 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:56:32.814568    4477 scope.go:122] "RemoveContainer" containerID="cb6e55d9badb01455dbb30f16c9da20ea07c3db38284d38cff68f4feeffae420"
Oct 30 22:56:32.814870 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:56:32.814851    4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb6e55d9badb01455dbb30f16c9da20ea07c3db38284d38cff68f4feeffae420\": container with ID starting with cb6e55d9badb01455dbb30f16c9da20ea07c3db38284d38cff68f4feeffae420 not found: ID does not exist" containerID="cb6e55d9badb01455dbb30f16c9da20ea07c3db38284d38cff68f4feeffae420"
Oct 30 22:56:32.814922 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:56:32.814879    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb6e55d9badb01455dbb30f16c9da20ea07c3db38284d38cff68f4feeffae420"} err="failed to get container status \"cb6e55d9badb01455dbb30f16c9da20ea07c3db38284d38cff68f4feeffae420\": rpc error: code = NotFound desc = could not find container \"cb6e55d9badb01455dbb30f16c9da20ea07c3db38284d38cff68f4feeffae420\": container with ID starting with cb6e55d9badb01455dbb30f16c9da20ea07c3db38284d38cff68f4feeffae420 not found: ID does not exist"
Oct 30 22:56:32.853540 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:56:32.853519    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-b03b33/auth-test-model-b03b33-kserve-d99bb5cb9-fzz5k"]
Oct 30 22:56:32.857522 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:56:32.857499    4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-b03b33/auth-test-model-b03b33-kserve-d99bb5cb9-fzz5k"]
Oct 30 22:56:34.195556 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:56:34.195526    4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae4a262e-38d8-493c-95f8-1bf666eff305" path="/var/lib/kubelet/pods/ae4a262e-38d8-493c-95f8-1bf666eff305/volumes"
Oct 30 22:56:34.945327 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:56:34.945292    4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-80ecb3/sub-test-model-80ecb3-kserve-9b4b44659-vnh9d" podUID="3c6d7321-d1ee-4023-ae2e-82dc126de49d" containerName="main" containerID="cri-o://399622a958c8788964713dbee6025d9f1914f857c983cdfdd4719b4839ee7671" gracePeriod=45
Oct 30 22:56:35.225141 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:56:35.225111    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-80ecb3/sub-test-model-80ecb3-kserve-9b4b44659-vnh9d"
Oct 30 22:56:35.315358 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:56:35.315328    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c6d7321-d1ee-4023-ae2e-82dc126de49d-home\" (UniqueName: \"kubernetes.io/empty-dir/3c6d7321-d1ee-4023-ae2e-82dc126de49d-home\") pod \"3c6d7321-d1ee-4023-ae2e-82dc126de49d\" (UID: \"3c6d7321-d1ee-4023-ae2e-82dc126de49d\") "
Oct 30 22:56:35.315358 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:56:35.315363    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c6d7321-d1ee-4023-ae2e-82dc126de49d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c6d7321-d1ee-4023-ae2e-82dc126de49d-dshm\") pod \"3c6d7321-d1ee-4023-ae2e-82dc126de49d\" (UID: \"3c6d7321-d1ee-4023-ae2e-82dc126de49d\") "
Oct 30 22:56:35.315600 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:56:35.315429    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c6d7321-d1ee-4023-ae2e-82dc126de49d-tls-certs\" (UniqueName: \"kubernetes.io/secret/3c6d7321-d1ee-4023-ae2e-82dc126de49d-tls-certs\") pod \"3c6d7321-d1ee-4023-ae2e-82dc126de49d\" (UID: \"3c6d7321-d1ee-4023-ae2e-82dc126de49d\") "
Oct 30 22:56:35.315600 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:56:35.315495    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c6d7321-d1ee-4023-ae2e-82dc126de49d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c6d7321-d1ee-4023-ae2e-82dc126de49d-tmp-dir\") pod \"3c6d7321-d1ee-4023-ae2e-82dc126de49d\" (UID: \"3c6d7321-d1ee-4023-ae2e-82dc126de49d\") "
Oct 30 22:56:35.315600 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:56:35.315574    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c6d7321-d1ee-4023-ae2e-82dc126de49d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c6d7321-d1ee-4023-ae2e-82dc126de49d-model-cache\") pod \"3c6d7321-d1ee-4023-ae2e-82dc126de49d\" (UID: \"3c6d7321-d1ee-4023-ae2e-82dc126de49d\") "
Oct 30 22:56:35.315754 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:56:35.315642    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c6d7321-d1ee-4023-ae2e-82dc126de49d-home" pod "3c6d7321-d1ee-4023-ae2e-82dc126de49d" (UID: "3c6d7321-d1ee-4023-ae2e-82dc126de49d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:56:35.315754 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:56:35.315689    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c6d7321-d1ee-4023-ae2e-82dc126de49d-kube-api-access-w9rtb\" (UniqueName: \"kubernetes.io/projected/3c6d7321-d1ee-4023-ae2e-82dc126de49d-kube-api-access-w9rtb\") pod \"3c6d7321-d1ee-4023-ae2e-82dc126de49d\" (UID: \"3c6d7321-d1ee-4023-ae2e-82dc126de49d\") "
Oct 30 22:56:35.315864 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:56:35.315840    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c6d7321-d1ee-4023-ae2e-82dc126de49d-tmp-dir" pod "3c6d7321-d1ee-4023-ae2e-82dc126de49d" (UID: "3c6d7321-d1ee-4023-ae2e-82dc126de49d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:56:35.315939 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:56:35.315917    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c6d7321-d1ee-4023-ae2e-82dc126de49d-model-cache" pod "3c6d7321-d1ee-4023-ae2e-82dc126de49d" (UID: "3c6d7321-d1ee-4023-ae2e-82dc126de49d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:56:35.316349 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:56:35.316319    4477 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c6d7321-d1ee-4023-ae2e-82dc126de49d-home\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:56:35.316349 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:56:35.316346    4477 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c6d7321-d1ee-4023-ae2e-82dc126de49d-tmp-dir\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:56:35.316523 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:56:35.316360    4477 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c6d7321-d1ee-4023-ae2e-82dc126de49d-model-cache\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:56:35.317842 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:56:35.317818    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c6d7321-d1ee-4023-ae2e-82dc126de49d-tls-certs" pod "3c6d7321-d1ee-4023-ae2e-82dc126de49d" (UID: "3c6d7321-d1ee-4023-ae2e-82dc126de49d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:56:35.317932 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:56:35.317815    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c6d7321-d1ee-4023-ae2e-82dc126de49d-kube-api-access-w9rtb" pod "3c6d7321-d1ee-4023-ae2e-82dc126de49d" (UID: "3c6d7321-d1ee-4023-ae2e-82dc126de49d"). InnerVolumeSpecName "kube-api-access-w9rtb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:56:35.317932 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:56:35.317824    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c6d7321-d1ee-4023-ae2e-82dc126de49d-dshm" pod "3c6d7321-d1ee-4023-ae2e-82dc126de49d" (UID: "3c6d7321-d1ee-4023-ae2e-82dc126de49d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:56:35.417734 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:56:35.417706    4477 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c6d7321-d1ee-4023-ae2e-82dc126de49d-dshm\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:56:35.417734 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:56:35.417733    4477 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c6d7321-d1ee-4023-ae2e-82dc126de49d-tls-certs\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:56:35.417903 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:56:35.417746    4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9rtb\" (UniqueName: \"kubernetes.io/projected/3c6d7321-d1ee-4023-ae2e-82dc126de49d-kube-api-access-w9rtb\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:56:35.875333 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:56:35.875291    4477 generic.go:356] "Generic (PLEG): container finished" podID="3c6d7321-d1ee-4023-ae2e-82dc126de49d" containerID="399622a958c8788964713dbee6025d9f1914f857c983cdfdd4719b4839ee7671" exitCode=0
Oct 30 22:56:35.875535 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:56:35.875345    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-80ecb3/sub-test-model-80ecb3-kserve-9b4b44659-vnh9d"
Oct 30 22:56:35.875535 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:56:35.875385    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-80ecb3/sub-test-model-80ecb3-kserve-9b4b44659-vnh9d" event={"ID":"3c6d7321-d1ee-4023-ae2e-82dc126de49d","Type":"ContainerDied","Data":"399622a958c8788964713dbee6025d9f1914f857c983cdfdd4719b4839ee7671"}
Oct 30 22:56:35.875535 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:56:35.875445    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-80ecb3/sub-test-model-80ecb3-kserve-9b4b44659-vnh9d" event={"ID":"3c6d7321-d1ee-4023-ae2e-82dc126de49d","Type":"ContainerDied","Data":"6140070c663f40a6509719594d51fa8e0f303a187fdb639c54019f2f7169a56d"}
Oct 30 22:56:35.875535 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:56:35.875471    4477 scope.go:122] "RemoveContainer" containerID="399622a958c8788964713dbee6025d9f1914f857c983cdfdd4719b4839ee7671"
Oct 30 22:56:35.888178 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:56:35.888160    4477 scope.go:122] "RemoveContainer" containerID="399622a958c8788964713dbee6025d9f1914f857c983cdfdd4719b4839ee7671"
Oct 30 22:56:35.888432 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:56:35.888409    4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"399622a958c8788964713dbee6025d9f1914f857c983cdfdd4719b4839ee7671\": container with ID starting with 399622a958c8788964713dbee6025d9f1914f857c983cdfdd4719b4839ee7671 not found: ID does not exist" containerID="399622a958c8788964713dbee6025d9f1914f857c983cdfdd4719b4839ee7671"
Oct 30 22:56:35.888529 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:56:35.888443    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"399622a958c8788964713dbee6025d9f1914f857c983cdfdd4719b4839ee7671"} err="failed to get container status \"399622a958c8788964713dbee6025d9f1914f857c983cdfdd4719b4839ee7671\": rpc error: code = NotFound desc = could not find container \"399622a958c8788964713dbee6025d9f1914f857c983cdfdd4719b4839ee7671\": container with ID starting with 399622a958c8788964713dbee6025d9f1914f857c983cdfdd4719b4839ee7671 not found: ID does not exist"
Oct 30 22:56:35.928257 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:56:35.928234    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-80ecb3/sub-test-model-80ecb3-kserve-9b4b44659-vnh9d"]
Oct 30 22:56:35.929240 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:56:35.929207    4477 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3c6d7321_d1ee_4023_ae2e_82dc126de49d.slice/crio-6140070c663f40a6509719594d51fa8e0f303a187fdb639c54019f2f7169a56d\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3c6d7321_d1ee_4023_ae2e_82dc126de49d.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 22:56:35.939216 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:56:35.939196    4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-80ecb3/sub-test-model-80ecb3-kserve-9b4b44659-vnh9d"]
Oct 30 22:56:36.229466 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:56:36.226013    4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c6d7321-d1ee-4023-ae2e-82dc126de49d" path="/var/lib/kubelet/pods/3c6d7321-d1ee-4023-ae2e-82dc126de49d/volumes"
Oct 30 22:56:56.266494 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:56:56.266458    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 22:56:57.062324 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:56:57.062193    4477 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jcf8l_015f218d-bf1f-4e7e-bb5d-b805af003246/ovn-acl-logging/0.log"
Oct 30 22:56:57.086346 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:56:57.086326    4477 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jcf8l_015f218d-bf1f-4e7e-bb5d-b805af003246/ovn-acl-logging/0.log"
Oct 30 22:57:26.268713 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:57:26.268671    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 22:57:56.267283 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:57:56.267238    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 22:58:04.558688 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:58:04.558643    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-23aab2/auth-test-model-23aab2-kserve-7f6f98db57-rbmt7"]
Oct 30 22:58:14.877881 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:58:14.877845    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6bz92"]
Oct 30 22:58:19.122256 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:58:19.122226    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-0ad7c5/distinct-2-w4-0ad7c5-kserve-77c5467fcb-skspk"]
Oct 30 22:58:19.136826 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:58:19.136799    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-0ad7c5/premium-w4-0ad7c5-kserve-7bbf78cd4-pm6wc"]
Oct 30 22:58:19.583755 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:58:19.583667    4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w1-23aab2/auth-test-model-23aab2-kserve-7f6f98db57-rbmt7" podUID="10316cee-ad81-4b83-acb8-6124dec5f00a" containerName="main" containerID="cri-o://796329acc1656f87e03bb6e3d8e41370e72e651a4e22d16104a3cbe72e680afa" gracePeriod=45
Oct 30 22:58:19.714688 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:58:19.714660    4477 generic.go:356] "Generic (PLEG): container finished" podID="10316cee-ad81-4b83-acb8-6124dec5f00a" containerID="796329acc1656f87e03bb6e3d8e41370e72e651a4e22d16104a3cbe72e680afa" exitCode=0
Oct 30 22:58:19.714796 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:58:19.714755    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-23aab2/auth-test-model-23aab2-kserve-7f6f98db57-rbmt7" event={"ID":"10316cee-ad81-4b83-acb8-6124dec5f00a","Type":"ContainerDied","Data":"796329acc1656f87e03bb6e3d8e41370e72e651a4e22d16104a3cbe72e680afa"}
Oct 30 22:58:19.863411 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:58:19.863384    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-23aab2/auth-test-model-23aab2-kserve-7f6f98db57-rbmt7"
Oct 30 22:58:19.999799 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:58:19.999759    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10316cee-ad81-4b83-acb8-6124dec5f00a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10316cee-ad81-4b83-acb8-6124dec5f00a-model-cache\") pod \"10316cee-ad81-4b83-acb8-6124dec5f00a\" (UID: \"10316cee-ad81-4b83-acb8-6124dec5f00a\") "
Oct 30 22:58:19.999979 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:58:19.999921    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10316cee-ad81-4b83-acb8-6124dec5f00a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10316cee-ad81-4b83-acb8-6124dec5f00a-tmp-dir\") pod \"10316cee-ad81-4b83-acb8-6124dec5f00a\" (UID: \"10316cee-ad81-4b83-acb8-6124dec5f00a\") "
Oct 30 22:58:20.000101 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:58:20.000084    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10316cee-ad81-4b83-acb8-6124dec5f00a-kube-api-access-rvfrq\" (UniqueName: \"kubernetes.io/projected/10316cee-ad81-4b83-acb8-6124dec5f00a-kube-api-access-rvfrq\") pod \"10316cee-ad81-4b83-acb8-6124dec5f00a\" (UID: \"10316cee-ad81-4b83-acb8-6124dec5f00a\") "
Oct 30 22:58:20.000184 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:58:20.000091    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10316cee-ad81-4b83-acb8-6124dec5f00a-model-cache" pod "10316cee-ad81-4b83-acb8-6124dec5f00a" (UID: "10316cee-ad81-4b83-acb8-6124dec5f00a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:58:20.000184 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:58:20.000139    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10316cee-ad81-4b83-acb8-6124dec5f00a-home\" (UniqueName: \"kubernetes.io/empty-dir/10316cee-ad81-4b83-acb8-6124dec5f00a-home\") pod \"10316cee-ad81-4b83-acb8-6124dec5f00a\" (UID: \"10316cee-ad81-4b83-acb8-6124dec5f00a\") "
Oct 30 22:58:20.000293 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:58:20.000187    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10316cee-ad81-4b83-acb8-6124dec5f00a-tmp-dir" pod "10316cee-ad81-4b83-acb8-6124dec5f00a" (UID: "10316cee-ad81-4b83-acb8-6124dec5f00a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:58:20.000293 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:58:20.000197    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10316cee-ad81-4b83-acb8-6124dec5f00a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/10316cee-ad81-4b83-acb8-6124dec5f00a-dshm\") pod \"10316cee-ad81-4b83-acb8-6124dec5f00a\" (UID: \"10316cee-ad81-4b83-acb8-6124dec5f00a\") "
Oct 30 22:58:20.000293 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:58:20.000236    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10316cee-ad81-4b83-acb8-6124dec5f00a-tls-certs\" (UniqueName: \"kubernetes.io/secret/10316cee-ad81-4b83-acb8-6124dec5f00a-tls-certs\") pod \"10316cee-ad81-4b83-acb8-6124dec5f00a\" (UID: \"10316cee-ad81-4b83-acb8-6124dec5f00a\") "
Oct 30 22:58:20.000439 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:58:20.000406    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10316cee-ad81-4b83-acb8-6124dec5f00a-home" pod "10316cee-ad81-4b83-acb8-6124dec5f00a" (UID: "10316cee-ad81-4b83-acb8-6124dec5f00a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:58:20.000830 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:58:20.000805    4477 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10316cee-ad81-4b83-acb8-6124dec5f00a-tmp-dir\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:58:20.000830 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:58:20.000830    4477 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10316cee-ad81-4b83-acb8-6124dec5f00a-home\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:58:20.001035 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:58:20.000843    4477 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10316cee-ad81-4b83-acb8-6124dec5f00a-model-cache\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:58:20.002559 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:58:20.002531    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10316cee-ad81-4b83-acb8-6124dec5f00a-dshm" pod "10316cee-ad81-4b83-acb8-6124dec5f00a" (UID: "10316cee-ad81-4b83-acb8-6124dec5f00a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:58:20.002754 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:58:20.002565    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10316cee-ad81-4b83-acb8-6124dec5f00a-kube-api-access-rvfrq" pod "10316cee-ad81-4b83-acb8-6124dec5f00a" (UID: "10316cee-ad81-4b83-acb8-6124dec5f00a"). InnerVolumeSpecName "kube-api-access-rvfrq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:58:20.002754 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:58:20.002582    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10316cee-ad81-4b83-acb8-6124dec5f00a-tls-certs" pod "10316cee-ad81-4b83-acb8-6124dec5f00a" (UID: "10316cee-ad81-4b83-acb8-6124dec5f00a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:58:20.102591 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:58:20.102513    4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvfrq\" (UniqueName: \"kubernetes.io/projected/10316cee-ad81-4b83-acb8-6124dec5f00a-kube-api-access-rvfrq\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:58:20.102591 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:58:20.102540    4477 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10316cee-ad81-4b83-acb8-6124dec5f00a-dshm\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:58:20.102591 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:58:20.102551    4477 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10316cee-ad81-4b83-acb8-6124dec5f00a-tls-certs\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:58:20.723588 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:58:20.723551    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-23aab2/auth-test-model-23aab2-kserve-7f6f98db57-rbmt7"
Oct 30 22:58:20.723588 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:58:20.723578    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-23aab2/auth-test-model-23aab2-kserve-7f6f98db57-rbmt7" event={"ID":"10316cee-ad81-4b83-acb8-6124dec5f00a","Type":"ContainerDied","Data":"e16171fd20876dfae636cf79d2dd3cd6b85f0cf76a633c074a0365eade19e8c4"}
Oct 30 22:58:20.724167 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:58:20.723629    4477 scope.go:122] "RemoveContainer" containerID="796329acc1656f87e03bb6e3d8e41370e72e651a4e22d16104a3cbe72e680afa"
Oct 30 22:58:20.777792 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:58:20.777767    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-23aab2/auth-test-model-23aab2-kserve-7f6f98db57-rbmt7"]
Oct 30 22:58:20.789511 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:58:20.789488    4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w1-23aab2/auth-test-model-23aab2-kserve-7f6f98db57-rbmt7"]
Oct 30 22:58:22.194164 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:58:22.194133    4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10316cee-ad81-4b83-acb8-6124dec5f00a" path="/var/lib/kubelet/pods/10316cee-ad81-4b83-acb8-6124dec5f00a/volumes"
Oct 30 22:58:24.815677 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:58:24.815646    4477 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-e3a7dd-6b6b8b4cfd-9gg8g_04451835-ece6-4466-8b50-71aaf24b4828/payload-processing/0.log"
Oct 30 22:58:26.272009 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:58:26.271967    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 22:58:34.145510 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:58:34.145472    4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-2-w4-0ad7c5-kserve-77c5467fcb-skspk" podUID="430bd879-2a89-4070-bb1b-9b9958a261b9" containerName="main" containerID="cri-o://8b0365cdaed18f133a96bf414c525bd03d2117181ce757a4b52a804d337c1d9b" gracePeriod=45
Oct 30 22:58:34.158312 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:58:34.158282    4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-0ad7c5/premium-w4-0ad7c5-kserve-7bbf78cd4-pm6wc" podUID="5c10371e-56e7-42ec-9ef8-0802e15cb2ff" containerName="main" containerID="cri-o://637557cda41a31bc56bb9a152f4d424881f7f5106ddff5d13c95faafcd60856e" gracePeriod=45
Oct 30 22:58:34.450912 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:58:34.449342    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-2-w4-0ad7c5-kserve-77c5467fcb-skspk"
Oct 30 22:58:34.467200 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:58:34.467174    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0ad7c5/premium-w4-0ad7c5-kserve-7bbf78cd4-pm6wc"
Oct 30 22:58:34.523419 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:58:34.523388    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/430bd879-2a89-4070-bb1b-9b9958a261b9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/430bd879-2a89-4070-bb1b-9b9958a261b9-tmp-dir\") pod \"430bd879-2a89-4070-bb1b-9b9958a261b9\" (UID: \"430bd879-2a89-4070-bb1b-9b9958a261b9\") "
Oct 30 22:58:34.523611 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:58:34.523492    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/430bd879-2a89-4070-bb1b-9b9958a261b9-tls-certs\" (UniqueName: \"kubernetes.io/secret/430bd879-2a89-4070-bb1b-9b9958a261b9-tls-certs\") pod \"430bd879-2a89-4070-bb1b-9b9958a261b9\" (UID: \"430bd879-2a89-4070-bb1b-9b9958a261b9\") "
Oct 30 22:58:34.523611 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:58:34.523538    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/430bd879-2a89-4070-bb1b-9b9958a261b9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/430bd879-2a89-4070-bb1b-9b9958a261b9-dshm\") pod \"430bd879-2a89-4070-bb1b-9b9958a261b9\" (UID: \"430bd879-2a89-4070-bb1b-9b9958a261b9\") "
Oct 30 22:58:34.523611 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:58:34.523580    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/430bd879-2a89-4070-bb1b-9b9958a261b9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/430bd879-2a89-4070-bb1b-9b9958a261b9-model-cache\") pod \"430bd879-2a89-4070-bb1b-9b9958a261b9\" (UID: \"430bd879-2a89-4070-bb1b-9b9958a261b9\") "
Oct 30 22:58:34.523813 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:58:34.523622    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/430bd879-2a89-4070-bb1b-9b9958a261b9-home\" (UniqueName: \"kubernetes.io/empty-dir/430bd879-2a89-4070-bb1b-9b9958a261b9-home\") pod \"430bd879-2a89-4070-bb1b-9b9958a261b9\" (UID: \"430bd879-2a89-4070-bb1b-9b9958a261b9\") "
Oct 30 22:58:34.523813 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:58:34.523664    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/430bd879-2a89-4070-bb1b-9b9958a261b9-kube-api-access-dgjh8\" (UniqueName: \"kubernetes.io/projected/430bd879-2a89-4070-bb1b-9b9958a261b9-kube-api-access-dgjh8\") pod \"430bd879-2a89-4070-bb1b-9b9958a261b9\" (UID: \"430bd879-2a89-4070-bb1b-9b9958a261b9\") "
Oct 30 22:58:34.523813 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:58:34.523760    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/430bd879-2a89-4070-bb1b-9b9958a261b9-tmp-dir" pod "430bd879-2a89-4070-bb1b-9b9958a261b9" (UID: "430bd879-2a89-4070-bb1b-9b9958a261b9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:58:34.523981 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:58:34.523903    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/430bd879-2a89-4070-bb1b-9b9958a261b9-model-cache" pod "430bd879-2a89-4070-bb1b-9b9958a261b9" (UID: "430bd879-2a89-4070-bb1b-9b9958a261b9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:58:34.524145 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:58:34.524115    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/430bd879-2a89-4070-bb1b-9b9958a261b9-home" pod "430bd879-2a89-4070-bb1b-9b9958a261b9" (UID: "430bd879-2a89-4070-bb1b-9b9958a261b9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:58:34.524551 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:58:34.524527    4477 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/430bd879-2a89-4070-bb1b-9b9958a261b9-tmp-dir\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:58:34.524667 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:58:34.524567    4477 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/430bd879-2a89-4070-bb1b-9b9958a261b9-model-cache\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:58:34.524667 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:58:34.524583    4477 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/430bd879-2a89-4070-bb1b-9b9958a261b9-home\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:58:34.526320 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:58:34.526282    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/430bd879-2a89-4070-bb1b-9b9958a261b9-kube-api-access-dgjh8" pod "430bd879-2a89-4070-bb1b-9b9958a261b9" (UID: "430bd879-2a89-4070-bb1b-9b9958a261b9"). InnerVolumeSpecName "kube-api-access-dgjh8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:58:34.526755 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:58:34.526732    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/430bd879-2a89-4070-bb1b-9b9958a261b9-dshm" pod "430bd879-2a89-4070-bb1b-9b9958a261b9" (UID: "430bd879-2a89-4070-bb1b-9b9958a261b9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:58:34.526864 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:58:34.526830    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/430bd879-2a89-4070-bb1b-9b9958a261b9-tls-certs" pod "430bd879-2a89-4070-bb1b-9b9958a261b9" (UID: "430bd879-2a89-4070-bb1b-9b9958a261b9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:58:34.625794 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:58:34.625754    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c10371e-56e7-42ec-9ef8-0802e15cb2ff-home\" (UniqueName: \"kubernetes.io/empty-dir/5c10371e-56e7-42ec-9ef8-0802e15cb2ff-home\") pod \"5c10371e-56e7-42ec-9ef8-0802e15cb2ff\" (UID: \"5c10371e-56e7-42ec-9ef8-0802e15cb2ff\") "
Oct 30 22:58:34.625983 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:58:34.625818    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c10371e-56e7-42ec-9ef8-0802e15cb2ff-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c10371e-56e7-42ec-9ef8-0802e15cb2ff-model-cache\") pod \"5c10371e-56e7-42ec-9ef8-0802e15cb2ff\" (UID: \"5c10371e-56e7-42ec-9ef8-0802e15cb2ff\") "
Oct 30 22:58:34.625983 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:58:34.625892    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c10371e-56e7-42ec-9ef8-0802e15cb2ff-kube-api-access-m9l6s\" (UniqueName: \"kubernetes.io/projected/5c10371e-56e7-42ec-9ef8-0802e15cb2ff-kube-api-access-m9l6s\") pod \"5c10371e-56e7-42ec-9ef8-0802e15cb2ff\" (UID: \"5c10371e-56e7-42ec-9ef8-0802e15cb2ff\") "
Oct 30 22:58:34.626143 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:58:34.626036    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c10371e-56e7-42ec-9ef8-0802e15cb2ff-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c10371e-56e7-42ec-9ef8-0802e15cb2ff-tmp-dir\") pod \"5c10371e-56e7-42ec-9ef8-0802e15cb2ff\" (UID: \"5c10371e-56e7-42ec-9ef8-0802e15cb2ff\") "
Oct 30 22:58:34.626143 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:58:34.626093    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c10371e-56e7-42ec-9ef8-0802e15cb2ff-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c10371e-56e7-42ec-9ef8-0802e15cb2ff-dshm\") pod \"5c10371e-56e7-42ec-9ef8-0802e15cb2ff\" (UID: \"5c10371e-56e7-42ec-9ef8-0802e15cb2ff\") "
Oct 30 22:58:34.626143 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:58:34.626128    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c10371e-56e7-42ec-9ef8-0802e15cb2ff-home" pod "5c10371e-56e7-42ec-9ef8-0802e15cb2ff" (UID: "5c10371e-56e7-42ec-9ef8-0802e15cb2ff"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:58:34.626319 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:58:34.626141    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c10371e-56e7-42ec-9ef8-0802e15cb2ff-model-cache" pod "5c10371e-56e7-42ec-9ef8-0802e15cb2ff" (UID: "5c10371e-56e7-42ec-9ef8-0802e15cb2ff"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:58:34.626319 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:58:34.626165    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c10371e-56e7-42ec-9ef8-0802e15cb2ff-tls-certs\" (UniqueName: \"kubernetes.io/secret/5c10371e-56e7-42ec-9ef8-0802e15cb2ff-tls-certs\") pod \"5c10371e-56e7-42ec-9ef8-0802e15cb2ff\" (UID: \"5c10371e-56e7-42ec-9ef8-0802e15cb2ff\") "
Oct 30 22:58:34.626437 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:58:34.626348    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c10371e-56e7-42ec-9ef8-0802e15cb2ff-tmp-dir" pod "5c10371e-56e7-42ec-9ef8-0802e15cb2ff" (UID: "5c10371e-56e7-42ec-9ef8-0802e15cb2ff"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:58:34.626782 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:58:34.626720    4477 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/430bd879-2a89-4070-bb1b-9b9958a261b9-tls-certs\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:58:34.626782 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:58:34.626746    4477 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/430bd879-2a89-4070-bb1b-9b9958a261b9-dshm\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:58:34.626782 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:58:34.626759    4477 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c10371e-56e7-42ec-9ef8-0802e15cb2ff-home\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:58:34.626782 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:58:34.626772    4477 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c10371e-56e7-42ec-9ef8-0802e15cb2ff-model-cache\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:58:34.626782 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:58:34.626785    4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgjh8\" (UniqueName: \"kubernetes.io/projected/430bd879-2a89-4070-bb1b-9b9958a261b9-kube-api-access-dgjh8\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:58:34.627178 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:58:34.626796    4477 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c10371e-56e7-42ec-9ef8-0802e15cb2ff-tmp-dir\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:58:34.628827 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:58:34.628723    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c10371e-56e7-42ec-9ef8-0802e15cb2ff-kube-api-access-m9l6s" pod "5c10371e-56e7-42ec-9ef8-0802e15cb2ff" (UID: "5c10371e-56e7-42ec-9ef8-0802e15cb2ff"). InnerVolumeSpecName "kube-api-access-m9l6s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:58:34.628827 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:58:34.628796    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c10371e-56e7-42ec-9ef8-0802e15cb2ff-tls-certs" pod "5c10371e-56e7-42ec-9ef8-0802e15cb2ff" (UID: "5c10371e-56e7-42ec-9ef8-0802e15cb2ff"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:58:34.629281 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:58:34.629261    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c10371e-56e7-42ec-9ef8-0802e15cb2ff-dshm" pod "5c10371e-56e7-42ec-9ef8-0802e15cb2ff" (UID: "5c10371e-56e7-42ec-9ef8-0802e15cb2ff"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:58:34.728553 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:58:34.728467    4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9l6s\" (UniqueName: \"kubernetes.io/projected/5c10371e-56e7-42ec-9ef8-0802e15cb2ff-kube-api-access-m9l6s\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:58:34.728553 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:58:34.728508    4477 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c10371e-56e7-42ec-9ef8-0802e15cb2ff-dshm\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:58:34.728553 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:58:34.728519    4477 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c10371e-56e7-42ec-9ef8-0802e15cb2ff-tls-certs\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 22:58:34.813684 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:58:34.813646    4477 generic.go:356] "Generic (PLEG): container finished" podID="430bd879-2a89-4070-bb1b-9b9958a261b9" containerID="8b0365cdaed18f133a96bf414c525bd03d2117181ce757a4b52a804d337c1d9b" exitCode=0
Oct 30 22:58:34.813852 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:58:34.813688    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-2-w4-0ad7c5-kserve-77c5467fcb-skspk"
Oct 30 22:58:34.813852 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:58:34.813728    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-2-w4-0ad7c5-kserve-77c5467fcb-skspk" event={"ID":"430bd879-2a89-4070-bb1b-9b9958a261b9","Type":"ContainerDied","Data":"8b0365cdaed18f133a96bf414c525bd03d2117181ce757a4b52a804d337c1d9b"}
Oct 30 22:58:34.813852 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:58:34.813773    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0ad7c5/distinct-2-w4-0ad7c5-kserve-77c5467fcb-skspk" event={"ID":"430bd879-2a89-4070-bb1b-9b9958a261b9","Type":"ContainerDied","Data":"01c1b3f250534bb730ea450723da9a7a5a93e59fd0ce2717374764e2b220b697"}
Oct 30 22:58:34.813852 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:58:34.813792    4477 scope.go:122] "RemoveContainer" containerID="8b0365cdaed18f133a96bf414c525bd03d2117181ce757a4b52a804d337c1d9b"
Oct 30 22:58:34.815314 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:58:34.815285    4477 generic.go:356] "Generic (PLEG): container finished" podID="5c10371e-56e7-42ec-9ef8-0802e15cb2ff" containerID="637557cda41a31bc56bb9a152f4d424881f7f5106ddff5d13c95faafcd60856e" exitCode=0
Oct 30 22:58:34.815462 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:58:34.815321    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0ad7c5/premium-w4-0ad7c5-kserve-7bbf78cd4-pm6wc"
Oct 30 22:58:34.815462 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:58:34.815355    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0ad7c5/premium-w4-0ad7c5-kserve-7bbf78cd4-pm6wc" event={"ID":"5c10371e-56e7-42ec-9ef8-0802e15cb2ff","Type":"ContainerDied","Data":"637557cda41a31bc56bb9a152f4d424881f7f5106ddff5d13c95faafcd60856e"}
Oct 30 22:58:34.815462 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:58:34.815394    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0ad7c5/premium-w4-0ad7c5-kserve-7bbf78cd4-pm6wc" event={"ID":"5c10371e-56e7-42ec-9ef8-0802e15cb2ff","Type":"ContainerDied","Data":"59bb12aad27a2bfa27356c911bdaec3827b96839dd378796c78baf753488cd25"}
Oct 30 22:58:34.825443 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:58:34.825425    4477 scope.go:122] "RemoveContainer" containerID="8b0365cdaed18f133a96bf414c525bd03d2117181ce757a4b52a804d337c1d9b"
Oct 30 22:58:34.825758 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:58:34.825733    4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b0365cdaed18f133a96bf414c525bd03d2117181ce757a4b52a804d337c1d9b\": container with ID starting with 8b0365cdaed18f133a96bf414c525bd03d2117181ce757a4b52a804d337c1d9b not found: ID does not exist" containerID="8b0365cdaed18f133a96bf414c525bd03d2117181ce757a4b52a804d337c1d9b"
Oct 30 22:58:34.825851 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:58:34.825768    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b0365cdaed18f133a96bf414c525bd03d2117181ce757a4b52a804d337c1d9b"} err="failed to get container status \"8b0365cdaed18f133a96bf414c525bd03d2117181ce757a4b52a804d337c1d9b\": rpc error: code = NotFound desc = could not find container \"8b0365cdaed18f133a96bf414c525bd03d2117181ce757a4b52a804d337c1d9b\": container with ID starting with 8b0365cdaed18f133a96bf414c525bd03d2117181ce757a4b52a804d337c1d9b not found: ID does not exist"
Oct 30 22:58:34.825851 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:58:34.825790    4477 scope.go:122] "RemoveContainer" containerID="637557cda41a31bc56bb9a152f4d424881f7f5106ddff5d13c95faafcd60856e"
Oct 30 22:58:34.837035 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:58:34.836708    4477 scope.go:122] "RemoveContainer" containerID="637557cda41a31bc56bb9a152f4d424881f7f5106ddff5d13c95faafcd60856e"
Oct 30 22:58:34.837035 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:58:34.837011    4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"637557cda41a31bc56bb9a152f4d424881f7f5106ddff5d13c95faafcd60856e\": container with ID starting with 637557cda41a31bc56bb9a152f4d424881f7f5106ddff5d13c95faafcd60856e not found: ID does not exist" containerID="637557cda41a31bc56bb9a152f4d424881f7f5106ddff5d13c95faafcd60856e"
Oct 30 22:58:34.837161 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:58:34.837055    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"637557cda41a31bc56bb9a152f4d424881f7f5106ddff5d13c95faafcd60856e"} err="failed to get container status \"637557cda41a31bc56bb9a152f4d424881f7f5106ddff5d13c95faafcd60856e\": rpc error: code = NotFound desc = could not find container \"637557cda41a31bc56bb9a152f4d424881f7f5106ddff5d13c95faafcd60856e\": container with ID starting with 637557cda41a31bc56bb9a152f4d424881f7f5106ddff5d13c95faafcd60856e not found: ID does not exist"
Oct 30 22:58:34.863319 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:58:34.863299    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-0ad7c5/premium-w4-0ad7c5-kserve-7bbf78cd4-pm6wc"]
Oct 30 22:58:34.867350 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:58:34.867322    4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-0ad7c5/premium-w4-0ad7c5-kserve-7bbf78cd4-pm6wc"]
Oct 30 22:58:34.901689 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:58:34.901668    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-0ad7c5/distinct-2-w4-0ad7c5-kserve-77c5467fcb-skspk"]
Oct 30 22:58:34.905866 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:58:34.905847    4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-0ad7c5/distinct-2-w4-0ad7c5-kserve-77c5467fcb-skspk"]
Oct 30 22:58:36.194466 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:58:36.194433    4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="430bd879-2a89-4070-bb1b-9b9958a261b9" path="/var/lib/kubelet/pods/430bd879-2a89-4070-bb1b-9b9958a261b9/volumes"
Oct 30 22:58:36.194891 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:58:36.194876    4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c10371e-56e7-42ec-9ef8-0802e15cb2ff" path="/var/lib/kubelet/pods/5c10371e-56e7-42ec-9ef8-0802e15cb2ff/volumes"
Oct 30 22:58:56.268006 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:58:56.267925    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 22:59:26.266939 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:59:26.266897    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 22:59:29.366438 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:59:29.366403    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6bz92"]
Oct 30 22:59:37.653894 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:59:37.653849    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-76d54a-69f954c4cd-kxw27"]
Oct 30 22:59:37.654470 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:59:37.654454    4477 cpu_manager.go:414] "containerMap: removing container" podUID="1e727c25-f834-4b00-aec6-9f3c7cb0b97b" containerName="main"
Oct 30 22:59:37.654516 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:59:37.654471    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e727c25-f834-4b00-aec6-9f3c7cb0b97b" containerName="main"
Oct 30 22:59:37.654516 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:59:37.654481    4477 cpu_manager.go:414] "containerMap: removing container" podUID="3c6d7321-d1ee-4023-ae2e-82dc126de49d" containerName="main"
Oct 30 22:59:37.654516 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:59:37.654486    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c6d7321-d1ee-4023-ae2e-82dc126de49d" containerName="main"
Oct 30 22:59:37.654516 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:59:37.654501    4477 cpu_manager.go:414] "containerMap: removing container" podUID="ae4a262e-38d8-493c-95f8-1bf666eff305" containerName="main"
Oct 30 22:59:37.654516 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:59:37.654506    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae4a262e-38d8-493c-95f8-1bf666eff305" containerName="main"
Oct 30 22:59:37.654516 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:59:37.654513    4477 cpu_manager.go:414] "containerMap: removing container" podUID="10316cee-ad81-4b83-acb8-6124dec5f00a" containerName="main"
Oct 30 22:59:37.654516 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:59:37.654517    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10316cee-ad81-4b83-acb8-6124dec5f00a" containerName="main"
Oct 30 22:59:37.654727 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:59:37.654526    4477 cpu_manager.go:414] "containerMap: removing container" podUID="5c10371e-56e7-42ec-9ef8-0802e15cb2ff" containerName="main"
Oct 30 22:59:37.654727 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:59:37.654530    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c10371e-56e7-42ec-9ef8-0802e15cb2ff" containerName="main"
Oct 30 22:59:37.654727 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:59:37.654559    4477 cpu_manager.go:414] "containerMap: removing container" podUID="430bd879-2a89-4070-bb1b-9b9958a261b9" containerName="main"
Oct 30 22:59:37.654727 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:59:37.654565    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="430bd879-2a89-4070-bb1b-9b9958a261b9" containerName="main"
Oct 30 22:59:37.654727 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:59:37.654655    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="430bd879-2a89-4070-bb1b-9b9958a261b9" containerName="main"
Oct 30 22:59:37.654727 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:59:37.654665    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e727c25-f834-4b00-aec6-9f3c7cb0b97b" containerName="main"
Oct 30 22:59:37.654727 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:59:37.654671    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c6d7321-d1ee-4023-ae2e-82dc126de49d" containerName="main"
Oct 30 22:59:37.654727 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:59:37.654678    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae4a262e-38d8-493c-95f8-1bf666eff305" containerName="main"
Oct 30 22:59:37.654727 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:59:37.654683    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="10316cee-ad81-4b83-acb8-6124dec5f00a" containerName="main"
Oct 30 22:59:37.654727 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:59:37.654688    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c10371e-56e7-42ec-9ef8-0802e15cb2ff" containerName="main"
Oct 30 22:59:37.658268 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:59:37.658247    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-76d54a-69f954c4cd-kxw27"
Oct 30 22:59:37.686268 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:59:37.686239    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-76d54a-serving-cert\""
Oct 30 22:59:37.691122 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:59:37.691094    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-76d54a-69f954c4cd-kxw27"]
Oct 30 22:59:37.692496 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:59:37.692464    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckzcj\" (UniqueName: \"kubernetes.io/projected/c17cee11-e2ac-469a-8759-c595e2e73191-kube-api-access-ckzcj\") pod \"maas-api-e2e-ipp-b-76d54a-69f954c4cd-kxw27\" (UID: \"c17cee11-e2ac-469a-8759-c595e2e73191\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-76d54a-69f954c4cd-kxw27"
Oct 30 22:59:37.692678 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:59:37.692653    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c17cee11-e2ac-469a-8759-c595e2e73191-metrics-certs\") pod \"maas-api-e2e-ipp-b-76d54a-69f954c4cd-kxw27\" (UID: \"c17cee11-e2ac-469a-8759-c595e2e73191\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-76d54a-69f954c4cd-kxw27"
Oct 30 22:59:37.692863 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:59:37.692844    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c17cee11-e2ac-469a-8759-c595e2e73191-maas-api-tls\") pod \"maas-api-e2e-ipp-b-76d54a-69f954c4cd-kxw27\" (UID: \"c17cee11-e2ac-469a-8759-c595e2e73191\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-76d54a-69f954c4cd-kxw27"
Oct 30 22:59:37.692915 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:59:37.692883    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c17cee11-e2ac-469a-8759-c595e2e73191-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-76d54a-69f954c4cd-kxw27\" (UID: \"c17cee11-e2ac-469a-8759-c595e2e73191\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-76d54a-69f954c4cd-kxw27"
Oct 30 22:59:37.794416 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:59:37.794386    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c17cee11-e2ac-469a-8759-c595e2e73191-metrics-certs\") pod \"maas-api-e2e-ipp-b-76d54a-69f954c4cd-kxw27\" (UID: \"c17cee11-e2ac-469a-8759-c595e2e73191\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-76d54a-69f954c4cd-kxw27"
Oct 30 22:59:37.794570 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:59:37.794553    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c17cee11-e2ac-469a-8759-c595e2e73191-maas-api-tls\") pod \"maas-api-e2e-ipp-b-76d54a-69f954c4cd-kxw27\" (UID: \"c17cee11-e2ac-469a-8759-c595e2e73191\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-76d54a-69f954c4cd-kxw27"
Oct 30 22:59:37.794644 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:59:37.794588    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c17cee11-e2ac-469a-8759-c595e2e73191-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-76d54a-69f954c4cd-kxw27\" (UID: \"c17cee11-e2ac-469a-8759-c595e2e73191\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-76d54a-69f954c4cd-kxw27"
Oct 30 22:59:37.794644 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:59:37.794633    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckzcj\" (UniqueName: \"kubernetes.io/projected/c17cee11-e2ac-469a-8759-c595e2e73191-kube-api-access-ckzcj\") pod \"maas-api-e2e-ipp-b-76d54a-69f954c4cd-kxw27\" (UID: \"c17cee11-e2ac-469a-8759-c595e2e73191\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-76d54a-69f954c4cd-kxw27"
Oct 30 22:59:37.794764 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:59:37.794708    4477 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-76d54a-serving-cert: secret "maas-api-e2e-ipp-b-76d54a-serving-cert" not found
Oct 30 22:59:37.794815 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:59:37.794790    4477 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c17cee11-e2ac-469a-8759-c595e2e73191-maas-api-tls podName:c17cee11-e2ac-469a-8759-c595e2e73191 nodeName:}" failed. No retries permitted until 2026-10-30 22:59:38.294769786 +0000 UTC m=+3162.220695970 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c17cee11-e2ac-469a-8759-c595e2e73191-maas-api-tls") pod "maas-api-e2e-ipp-b-76d54a-69f954c4cd-kxw27" (UID: "c17cee11-e2ac-469a-8759-c595e2e73191") : secret "maas-api-e2e-ipp-b-76d54a-serving-cert" not found
Oct 30 22:59:37.795273 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:59:37.795251    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c17cee11-e2ac-469a-8759-c595e2e73191-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-76d54a-69f954c4cd-kxw27\" (UID: \"c17cee11-e2ac-469a-8759-c595e2e73191\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-76d54a-69f954c4cd-kxw27"
Oct 30 22:59:37.796688 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:59:37.796669    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c17cee11-e2ac-469a-8759-c595e2e73191-metrics-certs\") pod \"maas-api-e2e-ipp-b-76d54a-69f954c4cd-kxw27\" (UID: \"c17cee11-e2ac-469a-8759-c595e2e73191\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-76d54a-69f954c4cd-kxw27"
Oct 30 22:59:37.813702 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:59:37.813679    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckzcj\" (UniqueName: \"kubernetes.io/projected/c17cee11-e2ac-469a-8759-c595e2e73191-kube-api-access-ckzcj\") pod \"maas-api-e2e-ipp-b-76d54a-69f954c4cd-kxw27\" (UID: \"c17cee11-e2ac-469a-8759-c595e2e73191\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-76d54a-69f954c4cd-kxw27"
Oct 30 22:59:38.302485 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:59:38.302451    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c17cee11-e2ac-469a-8759-c595e2e73191-maas-api-tls\") pod \"maas-api-e2e-ipp-b-76d54a-69f954c4cd-kxw27\" (UID: \"c17cee11-e2ac-469a-8759-c595e2e73191\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-76d54a-69f954c4cd-kxw27"
Oct 30 22:59:38.304847 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:59:38.304825    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c17cee11-e2ac-469a-8759-c595e2e73191-maas-api-tls\") pod \"maas-api-e2e-ipp-b-76d54a-69f954c4cd-kxw27\" (UID: \"c17cee11-e2ac-469a-8759-c595e2e73191\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-76d54a-69f954c4cd-kxw27"
Oct 30 22:59:38.572795 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:59:38.572702    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-76d54a-69f954c4cd-kxw27"
Oct 30 22:59:38.895125 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:59:38.889016    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-76d54a-69f954c4cd-kxw27"]
Oct 30 22:59:38.911274 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:59:38.911209    4477 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 22:59:39.404354 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:59:39.403754    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-76d54a-69f954c4cd-kxw27" event={"ID":"c17cee11-e2ac-469a-8759-c595e2e73191","Type":"ContainerStarted","Data":"22df9a1907c4403e844bc151fcb28a8adf1f1d51648831f046ef407efffeaa42"}
Oct 30 22:59:40.412442 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:59:40.412409    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-76d54a-69f954c4cd-kxw27" event={"ID":"c17cee11-e2ac-469a-8759-c595e2e73191","Type":"ContainerStarted","Data":"1777cad3317a72488d0f89aee6aebcee82b8fc969df44539839cefe007cd622d"}
Oct 30 22:59:40.412844 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:59:40.412568    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-76d54a-69f954c4cd-kxw27"
Oct 30 22:59:40.453151 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:59:40.453060    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-76d54a-69f954c4cd-kxw27" podStartSLOduration=3.042916266 podStartE2EDuration="3.453029021s" totalImagesPullingTime="410.112755ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:59:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:59:40.451407581 +0000 UTC m=+3164.377333787" watchObservedRunningTime="2026-10-30 22:59:40.453029021 +0000 UTC m=+3164.378955231"
Oct 30 22:59:46.423958 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:59:46.423929    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-76d54a-69f954c4cd-kxw27"
Oct 30 22:59:56.266791 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:59:56.266759    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 22:59:57.123849 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:59:57.123813    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-cdfa86-9b956fcf5-z6t2f"]
Oct 30 22:59:57.128398 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:59:57.128381    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cdfa86-9b956fcf5-z6t2f"
Oct 30 22:59:57.148663 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:59:57.148618    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-cdfa86-9b956fcf5-z6t2f"]
Oct 30 22:59:57.156663 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:59:57.156633    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-cdfa86-serving-cert\""
Oct 30 22:59:57.165425 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:59:57.165392    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/401f33fa-f0da-4809-af95-3f4d30ebc830-metrics-certs\") pod \"maas-api-e2e-ipp-b-cdfa86-9b956fcf5-z6t2f\" (UID: \"401f33fa-f0da-4809-af95-3f4d30ebc830\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cdfa86-9b956fcf5-z6t2f"
Oct 30 22:59:57.165558 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:59:57.165519    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/401f33fa-f0da-4809-af95-3f4d30ebc830-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-cdfa86-9b956fcf5-z6t2f\" (UID: \"401f33fa-f0da-4809-af95-3f4d30ebc830\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cdfa86-9b956fcf5-z6t2f"
Oct 30 22:59:57.165644 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:59:57.165625    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/401f33fa-f0da-4809-af95-3f4d30ebc830-maas-api-tls\") pod \"maas-api-e2e-ipp-b-cdfa86-9b956fcf5-z6t2f\" (UID: \"401f33fa-f0da-4809-af95-3f4d30ebc830\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cdfa86-9b956fcf5-z6t2f"
Oct 30 22:59:57.165819 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:59:57.165798    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kq8f\" (UniqueName: \"kubernetes.io/projected/401f33fa-f0da-4809-af95-3f4d30ebc830-kube-api-access-7kq8f\") pod \"maas-api-e2e-ipp-b-cdfa86-9b956fcf5-z6t2f\" (UID: \"401f33fa-f0da-4809-af95-3f4d30ebc830\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cdfa86-9b956fcf5-z6t2f"
Oct 30 22:59:57.267408 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:59:57.267377    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/401f33fa-f0da-4809-af95-3f4d30ebc830-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-cdfa86-9b956fcf5-z6t2f\" (UID: \"401f33fa-f0da-4809-af95-3f4d30ebc830\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cdfa86-9b956fcf5-z6t2f"
Oct 30 22:59:57.267847 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:59:57.267446    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/401f33fa-f0da-4809-af95-3f4d30ebc830-maas-api-tls\") pod \"maas-api-e2e-ipp-b-cdfa86-9b956fcf5-z6t2f\" (UID: \"401f33fa-f0da-4809-af95-3f4d30ebc830\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cdfa86-9b956fcf5-z6t2f"
Oct 30 22:59:57.267847 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:59:57.267592    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kq8f\" (UniqueName: \"kubernetes.io/projected/401f33fa-f0da-4809-af95-3f4d30ebc830-kube-api-access-7kq8f\") pod \"maas-api-e2e-ipp-b-cdfa86-9b956fcf5-z6t2f\" (UID: \"401f33fa-f0da-4809-af95-3f4d30ebc830\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cdfa86-9b956fcf5-z6t2f"
Oct 30 22:59:57.267847 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:59:57.267603    4477 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-cdfa86-serving-cert: secret "maas-api-e2e-ipp-b-cdfa86-serving-cert" not found
Oct 30 22:59:57.267847 ip-10-0-130-77 kubenswrapper[4477]: E1030 22:59:57.267682    4477 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/401f33fa-f0da-4809-af95-3f4d30ebc830-maas-api-tls podName:401f33fa-f0da-4809-af95-3f4d30ebc830 nodeName:}" failed. No retries permitted until 2026-10-30 22:59:57.7676614 +0000 UTC m=+3181.693587811 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/401f33fa-f0da-4809-af95-3f4d30ebc830-maas-api-tls") pod "maas-api-e2e-ipp-b-cdfa86-9b956fcf5-z6t2f" (UID: "401f33fa-f0da-4809-af95-3f4d30ebc830") : secret "maas-api-e2e-ipp-b-cdfa86-serving-cert" not found
Oct 30 22:59:57.267847 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:59:57.267762    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/401f33fa-f0da-4809-af95-3f4d30ebc830-metrics-certs\") pod \"maas-api-e2e-ipp-b-cdfa86-9b956fcf5-z6t2f\" (UID: \"401f33fa-f0da-4809-af95-3f4d30ebc830\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cdfa86-9b956fcf5-z6t2f"
Oct 30 22:59:57.268203 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:59:57.268184    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/401f33fa-f0da-4809-af95-3f4d30ebc830-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-cdfa86-9b956fcf5-z6t2f\" (UID: \"401f33fa-f0da-4809-af95-3f4d30ebc830\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cdfa86-9b956fcf5-z6t2f"
Oct 30 22:59:57.270146 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:59:57.270120    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/401f33fa-f0da-4809-af95-3f4d30ebc830-metrics-certs\") pod \"maas-api-e2e-ipp-b-cdfa86-9b956fcf5-z6t2f\" (UID: \"401f33fa-f0da-4809-af95-3f4d30ebc830\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cdfa86-9b956fcf5-z6t2f"
Oct 30 22:59:57.286716 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:59:57.286688    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kq8f\" (UniqueName: \"kubernetes.io/projected/401f33fa-f0da-4809-af95-3f4d30ebc830-kube-api-access-7kq8f\") pod \"maas-api-e2e-ipp-b-cdfa86-9b956fcf5-z6t2f\" (UID: \"401f33fa-f0da-4809-af95-3f4d30ebc830\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cdfa86-9b956fcf5-z6t2f"
Oct 30 22:59:57.773539 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:59:57.773506    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/401f33fa-f0da-4809-af95-3f4d30ebc830-maas-api-tls\") pod \"maas-api-e2e-ipp-b-cdfa86-9b956fcf5-z6t2f\" (UID: \"401f33fa-f0da-4809-af95-3f4d30ebc830\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cdfa86-9b956fcf5-z6t2f"
Oct 30 22:59:57.775943 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:59:57.775917    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/401f33fa-f0da-4809-af95-3f4d30ebc830-maas-api-tls\") pod \"maas-api-e2e-ipp-b-cdfa86-9b956fcf5-z6t2f\" (UID: \"401f33fa-f0da-4809-af95-3f4d30ebc830\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cdfa86-9b956fcf5-z6t2f"
Oct 30 22:59:58.045176 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:59:58.045090    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cdfa86-9b956fcf5-z6t2f"
Oct 30 22:59:58.400428 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:59:58.391139    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-cdfa86-9b956fcf5-z6t2f"]
Oct 30 22:59:58.407231 ip-10-0-130-77 kubenswrapper[4477]: W1030 22:59:58.407192    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod401f33fa_f0da_4809_af95_3f4d30ebc830.slice/crio-457f812cd120938b8cbb9f69c278cdd6029fc0564684a414722e5f4dd5c285f6 WatchSource:0}: Error finding container 457f812cd120938b8cbb9f69c278cdd6029fc0564684a414722e5f4dd5c285f6: Status 404 returned error can't find the container with id 457f812cd120938b8cbb9f69c278cdd6029fc0564684a414722e5f4dd5c285f6
Oct 30 22:59:58.527461 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:59:58.527429    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cdfa86-9b956fcf5-z6t2f" event={"ID":"401f33fa-f0da-4809-af95-3f4d30ebc830","Type":"ContainerStarted","Data":"457f812cd120938b8cbb9f69c278cdd6029fc0564684a414722e5f4dd5c285f6"}
Oct 30 22:59:59.535287 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:59:59.535249    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cdfa86-9b956fcf5-z6t2f" event={"ID":"401f33fa-f0da-4809-af95-3f4d30ebc830","Type":"ContainerStarted","Data":"ea3ef4b8d735d2d2c6ad8b31d5bfa268a0b07aea6257484f1567a28e43c09b21"}
Oct 30 22:59:59.535773 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:59:59.535418    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cdfa86-9b956fcf5-z6t2f"
Oct 30 22:59:59.577081 ip-10-0-130-77 kubenswrapper[4477]: I1030 22:59:59.577018    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cdfa86-9b956fcf5-z6t2f" podStartSLOduration=2.275204516 podStartE2EDuration="2.577005212s" totalImagesPullingTime="301.800696ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:59:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:59:59.576007547 +0000 UTC m=+3183.501933753" watchObservedRunningTime="2026-10-30 22:59:59.577005212 +0000 UTC m=+3183.502931418"
Oct 30 23:00:00.284227 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:00.284194    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f88979-29890020-ngbtw"]
Oct 30 23:00:00.289190 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:00.289162    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-76d54a-29890020-dv4dj"]
Oct 30 23:00:00.289445 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:00.289292    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f88979-29890020-ngbtw"
Oct 30 23:00:00.298744 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:00.298363    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f88979-29890020-ngbtw"]
Oct 30 23:00:00.298744 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:00.298473    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-76d54a-29890020-dv4dj"
Oct 30 23:00:00.304984 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:00.304965    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-76d54a-29890020-dv4dj"]
Oct 30 23:00:00.332059 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:00.332001    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 23:00:00.411221 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:00.411073    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkfz8\" (UniqueName: \"kubernetes.io/projected/a910f2eb-f5b8-45fc-a827-103d116454f9-kube-api-access-lkfz8\") pod \"maas-api-key-cleanup-e2e-ipp-b-76d54a-29890020-dv4dj\" (UID: \"a910f2eb-f5b8-45fc-a827-103d116454f9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-76d54a-29890020-dv4dj"
Oct 30 23:00:00.411406 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:00.411227    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqvfr\" (UniqueName: \"kubernetes.io/projected/cdcfe2c6-edf0-42a3-ac07-434812bedaa5-kube-api-access-jqvfr\") pod \"maas-api-key-cleanup-e2e-ipp-a-f88979-29890020-ngbtw\" (UID: \"cdcfe2c6-edf0-42a3-ac07-434812bedaa5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f88979-29890020-ngbtw"
Oct 30 23:00:00.411406 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:00.411259    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a910f2eb-f5b8-45fc-a827-103d116454f9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-76d54a-29890020-dv4dj\" (UID: \"a910f2eb-f5b8-45fc-a827-103d116454f9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-76d54a-29890020-dv4dj"
Oct 30 23:00:00.411406 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:00.411294    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdcfe2c6-edf0-42a3-ac07-434812bedaa5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-f88979-29890020-ngbtw\" (UID: \"cdcfe2c6-edf0-42a3-ac07-434812bedaa5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f88979-29890020-ngbtw"
Oct 30 23:00:00.512669 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:00.512634    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqvfr\" (UniqueName: \"kubernetes.io/projected/cdcfe2c6-edf0-42a3-ac07-434812bedaa5-kube-api-access-jqvfr\") pod \"maas-api-key-cleanup-e2e-ipp-a-f88979-29890020-ngbtw\" (UID: \"cdcfe2c6-edf0-42a3-ac07-434812bedaa5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f88979-29890020-ngbtw"
Oct 30 23:00:00.512838 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:00.512675    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a910f2eb-f5b8-45fc-a827-103d116454f9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-76d54a-29890020-dv4dj\" (UID: \"a910f2eb-f5b8-45fc-a827-103d116454f9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-76d54a-29890020-dv4dj"
Oct 30 23:00:00.512838 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:00.512716    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdcfe2c6-edf0-42a3-ac07-434812bedaa5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-f88979-29890020-ngbtw\" (UID: \"cdcfe2c6-edf0-42a3-ac07-434812bedaa5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f88979-29890020-ngbtw"
Oct 30 23:00:00.512913 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:00.512873    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkfz8\" (UniqueName: \"kubernetes.io/projected/a910f2eb-f5b8-45fc-a827-103d116454f9-kube-api-access-lkfz8\") pod \"maas-api-key-cleanup-e2e-ipp-b-76d54a-29890020-dv4dj\" (UID: \"a910f2eb-f5b8-45fc-a827-103d116454f9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-76d54a-29890020-dv4dj"
Oct 30 23:00:00.513609 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:00.513584    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdcfe2c6-edf0-42a3-ac07-434812bedaa5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-f88979-29890020-ngbtw\" (UID: \"cdcfe2c6-edf0-42a3-ac07-434812bedaa5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f88979-29890020-ngbtw"
Oct 30 23:00:00.513676 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:00.513584    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a910f2eb-f5b8-45fc-a827-103d116454f9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-76d54a-29890020-dv4dj\" (UID: \"a910f2eb-f5b8-45fc-a827-103d116454f9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-76d54a-29890020-dv4dj"
Oct 30 23:00:00.534720 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:00.534653    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkfz8\" (UniqueName: \"kubernetes.io/projected/a910f2eb-f5b8-45fc-a827-103d116454f9-kube-api-access-lkfz8\") pod \"maas-api-key-cleanup-e2e-ipp-b-76d54a-29890020-dv4dj\" (UID: \"a910f2eb-f5b8-45fc-a827-103d116454f9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-76d54a-29890020-dv4dj"
Oct 30 23:00:00.534846 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:00.534752    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqvfr\" (UniqueName: \"kubernetes.io/projected/cdcfe2c6-edf0-42a3-ac07-434812bedaa5-kube-api-access-jqvfr\") pod \"maas-api-key-cleanup-e2e-ipp-a-f88979-29890020-ngbtw\" (UID: \"cdcfe2c6-edf0-42a3-ac07-434812bedaa5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f88979-29890020-ngbtw"
Oct 30 23:00:00.618718 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:00.618688    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f88979-29890020-ngbtw"
Oct 30 23:00:00.624781 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:00.624760    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-76d54a-29890020-dv4dj"
Oct 30 23:00:00.919697 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:00.919661    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f88979-29890020-ngbtw"]
Oct 30 23:00:00.923130 ip-10-0-130-77 kubenswrapper[4477]: W1030 23:00:00.923097    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdcfe2c6_edf0_42a3_ac07_434812bedaa5.slice/crio-a315f72dcd6eeb414e748735bd0e847634de786f7943db45f0b927587f26cd94 WatchSource:0}: Error finding container a315f72dcd6eeb414e748735bd0e847634de786f7943db45f0b927587f26cd94: Status 404 returned error can't find the container with id a315f72dcd6eeb414e748735bd0e847634de786f7943db45f0b927587f26cd94
Oct 30 23:00:00.933656 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:00.933632    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-76d54a-29890020-dv4dj"]
Oct 30 23:00:00.937415 ip-10-0-130-77 kubenswrapper[4477]: W1030 23:00:00.937390    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda910f2eb_f5b8_45fc_a827_103d116454f9.slice/crio-b74cf2ff553441bc04ebc9be491dab5beed7466f062fc0c06e25dd25adf1607e WatchSource:0}: Error finding container b74cf2ff553441bc04ebc9be491dab5beed7466f062fc0c06e25dd25adf1607e: Status 404 returned error can't find the container with id b74cf2ff553441bc04ebc9be491dab5beed7466f062fc0c06e25dd25adf1607e
Oct 30 23:00:01.549520 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:01.549443    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f88979-29890020-ngbtw" event={"ID":"cdcfe2c6-edf0-42a3-ac07-434812bedaa5","Type":"ContainerStarted","Data":"7c9f74de371d637e7ed530286e7504e723401c28069822b18a3f19a516795f37"}
Oct 30 23:00:01.549520 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:01.549496    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f88979-29890020-ngbtw" event={"ID":"cdcfe2c6-edf0-42a3-ac07-434812bedaa5","Type":"ContainerStarted","Data":"a315f72dcd6eeb414e748735bd0e847634de786f7943db45f0b927587f26cd94"}
Oct 30 23:00:01.551054 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:01.551017    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-76d54a-29890020-dv4dj" event={"ID":"a910f2eb-f5b8-45fc-a827-103d116454f9","Type":"ContainerStarted","Data":"0d39e72810e3588115b08be882632429354717b7af19bf8d8a00cdf4db501f0a"}
Oct 30 23:00:01.551155 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:01.551092    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-76d54a-29890020-dv4dj" event={"ID":"a910f2eb-f5b8-45fc-a827-103d116454f9","Type":"ContainerStarted","Data":"b74cf2ff553441bc04ebc9be491dab5beed7466f062fc0c06e25dd25adf1607e"}
Oct 30 23:00:01.587792 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:01.587739    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f88979-29890020-ngbtw" podStartSLOduration=1.587725528 podStartE2EDuration="1.587725528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:00:01.585583003 +0000 UTC m=+3185.511509226" watchObservedRunningTime="2026-10-30 23:00:01.587725528 +0000 UTC m=+3185.513651733"
Oct 30 23:00:01.627076 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:01.627018    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-76d54a-29890020-dv4dj" podStartSLOduration=1.6270052229999998 podStartE2EDuration="1.627005223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:00:01.623442765 +0000 UTC m=+3185.549368990" watchObservedRunningTime="2026-10-30 23:00:01.627005223 +0000 UTC m=+3185.552931429"
Oct 30 23:00:02.559811 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:02.559774    4477 generic.go:356] "Generic (PLEG): container finished" podID="cdcfe2c6-edf0-42a3-ac07-434812bedaa5" containerID="7c9f74de371d637e7ed530286e7504e723401c28069822b18a3f19a516795f37" exitCode=60
Oct 30 23:00:02.560005 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:02.559851    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f88979-29890020-ngbtw" event={"ID":"cdcfe2c6-edf0-42a3-ac07-434812bedaa5","Type":"ContainerDied","Data":"7c9f74de371d637e7ed530286e7504e723401c28069822b18a3f19a516795f37"}
Oct 30 23:00:02.560151 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:02.560123    4477 scope.go:122] "RemoveContainer" containerID="7c9f74de371d637e7ed530286e7504e723401c28069822b18a3f19a516795f37"
Oct 30 23:00:02.561895 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:02.561867    4477 generic.go:356] "Generic (PLEG): container finished" podID="a910f2eb-f5b8-45fc-a827-103d116454f9" containerID="0d39e72810e3588115b08be882632429354717b7af19bf8d8a00cdf4db501f0a" exitCode=60
Oct 30 23:00:02.562159 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:02.561897    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-76d54a-29890020-dv4dj" event={"ID":"a910f2eb-f5b8-45fc-a827-103d116454f9","Type":"ContainerDied","Data":"0d39e72810e3588115b08be882632429354717b7af19bf8d8a00cdf4db501f0a"}
Oct 30 23:00:02.562285 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:02.562246    4477 scope.go:122] "RemoveContainer" containerID="0d39e72810e3588115b08be882632429354717b7af19bf8d8a00cdf4db501f0a"
Oct 30 23:00:03.569497 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:03.569468    4477 generic.go:356] "Generic (PLEG): container finished" podID="cdcfe2c6-edf0-42a3-ac07-434812bedaa5" containerID="e4222498307eeb32298453f7fd68480abf6e9fb3f7a5162d509cf379b02b7ed3" exitCode=60
Oct 30 23:00:03.569886 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:03.569501    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f88979-29890020-ngbtw" event={"ID":"cdcfe2c6-edf0-42a3-ac07-434812bedaa5","Type":"ContainerDied","Data":"e4222498307eeb32298453f7fd68480abf6e9fb3f7a5162d509cf379b02b7ed3"}
Oct 30 23:00:03.569886 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:03.569564    4477 scope.go:122] "RemoveContainer" containerID="7c9f74de371d637e7ed530286e7504e723401c28069822b18a3f19a516795f37"
Oct 30 23:00:03.569886 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:03.569795    4477 scope.go:122] "RemoveContainer" containerID="e4222498307eeb32298453f7fd68480abf6e9fb3f7a5162d509cf379b02b7ed3"
Oct 30 23:00:03.570117 ip-10-0-130-77 kubenswrapper[4477]: E1030 23:00:03.570093    4477 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-f88979-29890020-ngbtw_odh-ai-gateway-infra(cdcfe2c6-edf0-42a3-ac07-434812bedaa5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f88979-29890020-ngbtw" podUID="cdcfe2c6-edf0-42a3-ac07-434812bedaa5"
Oct 30 23:00:03.571920 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:03.571898    4477 generic.go:356] "Generic (PLEG): container finished" podID="a910f2eb-f5b8-45fc-a827-103d116454f9" containerID="1b494d04bd0230bdc2420d3afbb6319a6d06f958123287a5fd08ec6cc708c09f" exitCode=60
Oct 30 23:00:03.572004 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:03.571927    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-76d54a-29890020-dv4dj" event={"ID":"a910f2eb-f5b8-45fc-a827-103d116454f9","Type":"ContainerDied","Data":"1b494d04bd0230bdc2420d3afbb6319a6d06f958123287a5fd08ec6cc708c09f"}
Oct 30 23:00:03.572235 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:03.572221    4477 scope.go:122] "RemoveContainer" containerID="1b494d04bd0230bdc2420d3afbb6319a6d06f958123287a5fd08ec6cc708c09f"
Oct 30 23:00:03.572471 ip-10-0-130-77 kubenswrapper[4477]: E1030 23:00:03.572455    4477 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-76d54a-29890020-dv4dj_odh-ai-gateway-infra(a910f2eb-f5b8-45fc-a827-103d116454f9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-76d54a-29890020-dv4dj" podUID="a910f2eb-f5b8-45fc-a827-103d116454f9"
Oct 30 23:00:03.582278 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:03.582261    4477 scope.go:122] "RemoveContainer" containerID="0d39e72810e3588115b08be882632429354717b7af19bf8d8a00cdf4db501f0a"
Oct 30 23:00:04.579272 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:04.579245    4477 scope.go:122] "RemoveContainer" containerID="1b494d04bd0230bdc2420d3afbb6319a6d06f958123287a5fd08ec6cc708c09f"
Oct 30 23:00:04.579694 ip-10-0-130-77 kubenswrapper[4477]: E1030 23:00:04.579469    4477 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-76d54a-29890020-dv4dj_odh-ai-gateway-infra(a910f2eb-f5b8-45fc-a827-103d116454f9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-76d54a-29890020-dv4dj" podUID="a910f2eb-f5b8-45fc-a827-103d116454f9"
Oct 30 23:00:04.581068 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:04.581030    4477 scope.go:122] "RemoveContainer" containerID="e4222498307eeb32298453f7fd68480abf6e9fb3f7a5162d509cf379b02b7ed3"
Oct 30 23:00:04.581264 ip-10-0-130-77 kubenswrapper[4477]: E1030 23:00:04.581246    4477 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-f88979-29890020-ngbtw_odh-ai-gateway-infra(cdcfe2c6-edf0-42a3-ac07-434812bedaa5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f88979-29890020-ngbtw" podUID="cdcfe2c6-edf0-42a3-ac07-434812bedaa5"
Oct 30 23:00:05.547264 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:05.547234    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cdfa86-9b956fcf5-z6t2f"
Oct 30 23:00:14.187397 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:14.187308    4477 scope.go:122] "RemoveContainer" containerID="1b494d04bd0230bdc2420d3afbb6319a6d06f958123287a5fd08ec6cc708c09f"
Oct 30 23:00:14.642120 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:14.642082    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-76d54a-29890020-dv4dj" event={"ID":"a910f2eb-f5b8-45fc-a827-103d116454f9","Type":"ContainerStarted","Data":"64ec9470be9a199c70d84ae81672637b4d89f0ffe935df38e8f2e983b43359c2"}
Oct 30 23:00:15.649942 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:15.649908    4477 generic.go:356] "Generic (PLEG): container finished" podID="a910f2eb-f5b8-45fc-a827-103d116454f9" containerID="64ec9470be9a199c70d84ae81672637b4d89f0ffe935df38e8f2e983b43359c2" exitCode=60
Oct 30 23:00:15.650423 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:15.649980    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-76d54a-29890020-dv4dj" event={"ID":"a910f2eb-f5b8-45fc-a827-103d116454f9","Type":"ContainerDied","Data":"64ec9470be9a199c70d84ae81672637b4d89f0ffe935df38e8f2e983b43359c2"}
Oct 30 23:00:15.650423 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:15.650063    4477 scope.go:122] "RemoveContainer" containerID="1b494d04bd0230bdc2420d3afbb6319a6d06f958123287a5fd08ec6cc708c09f"
Oct 30 23:00:15.650423 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:15.650393    4477 scope.go:122] "RemoveContainer" containerID="64ec9470be9a199c70d84ae81672637b4d89f0ffe935df38e8f2e983b43359c2"
Oct 30 23:00:15.650711 ip-10-0-130-77 kubenswrapper[4477]: E1030 23:00:15.650687    4477 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-76d54a-29890020-dv4dj_odh-ai-gateway-infra(a910f2eb-f5b8-45fc-a827-103d116454f9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-76d54a-29890020-dv4dj" podUID="a910f2eb-f5b8-45fc-a827-103d116454f9"
Oct 30 23:00:15.690522 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:15.690497    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-76d54a-29890020-dv4dj"]
Oct 30 23:00:15.898263 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:15.898228    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e3a7dd-866fc6f4cb-gqb2f"]
Oct 30 23:00:15.898452 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:15.898430    4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e3a7dd-866fc6f4cb-gqb2f" podUID="dbb8248f-a7c8-4a23-80cd-ba28e98da140" containerName="maas-api" containerID="cri-o://272fddee0f30375f296ce1a617eb22a0752559e957bf069cf62e04e9ed249685" gracePeriod=30
Oct 30 23:00:16.099318 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:16.098123    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e3a7dd-6b6b8b4cfd-9gg8g"]
Oct 30 23:00:16.099318 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:16.098357    4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-e3a7dd-6b6b8b4cfd-9gg8g" podUID="04451835-ece6-4466-8b50-71aaf24b4828" containerName="payload-processing" containerID="cri-o://0b8f3f1cdbda169c2a7c2ea51bdf5003cccf2428fc9b6df4473f8953414805b5" gracePeriod=30
Oct 30 23:00:16.111593 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:16.111541    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e3a7dd-fc4fb5599-mg28w"]
Oct 30 23:00:16.111861 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:16.111800    4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e3a7dd-fc4fb5599-mg28w" podUID="c178db77-b489-434c-99e3-c7934518a7f8" containerName="payload-pre-processing" containerID="cri-o://43ef41e7bf29dfd1a3369830f718bbb11eedecd6381a8f283a099f8f5ab4ce3a" gracePeriod=30
Oct 30 23:00:16.280091 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:16.279248    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e3a7dd-866fc6f4cb-gqb2f"
Oct 30 23:00:16.335380 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:16.332010    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dbb8248f-a7c8-4a23-80cd-ba28e98da140-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dbb8248f-a7c8-4a23-80cd-ba28e98da140-maas-api-tls\") pod \"dbb8248f-a7c8-4a23-80cd-ba28e98da140\" (UID: \"dbb8248f-a7c8-4a23-80cd-ba28e98da140\") "
Oct 30 23:00:16.335380 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:16.332885    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dbb8248f-a7c8-4a23-80cd-ba28e98da140-metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbb8248f-a7c8-4a23-80cd-ba28e98da140-metrics-certs\") pod \"dbb8248f-a7c8-4a23-80cd-ba28e98da140\" (UID: \"dbb8248f-a7c8-4a23-80cd-ba28e98da140\") "
Oct 30 23:00:16.335380 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:16.332989    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbb8248f-a7c8-4a23-80cd-ba28e98da140-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbb8248f-a7c8-4a23-80cd-ba28e98da140-openshift-service-ca\") pod \"dbb8248f-a7c8-4a23-80cd-ba28e98da140\" (UID: \"dbb8248f-a7c8-4a23-80cd-ba28e98da140\") "
Oct 30 23:00:16.335380 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:16.333126    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbb8248f-a7c8-4a23-80cd-ba28e98da140-kube-api-access-mtrtz\" (UniqueName: \"kubernetes.io/projected/dbb8248f-a7c8-4a23-80cd-ba28e98da140-kube-api-access-mtrtz\") pod \"dbb8248f-a7c8-4a23-80cd-ba28e98da140\" (UID: \"dbb8248f-a7c8-4a23-80cd-ba28e98da140\") "
Oct 30 23:00:16.335380 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:16.335130    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbb8248f-a7c8-4a23-80cd-ba28e98da140-openshift-service-ca" pod "dbb8248f-a7c8-4a23-80cd-ba28e98da140" (UID: "dbb8248f-a7c8-4a23-80cd-ba28e98da140"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:00:16.339382 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:16.339339    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbb8248f-a7c8-4a23-80cd-ba28e98da140-metrics-certs" pod "dbb8248f-a7c8-4a23-80cd-ba28e98da140" (UID: "dbb8248f-a7c8-4a23-80cd-ba28e98da140"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:00:16.347897 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:16.347865    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbb8248f-a7c8-4a23-80cd-ba28e98da140-maas-api-tls" pod "dbb8248f-a7c8-4a23-80cd-ba28e98da140" (UID: "dbb8248f-a7c8-4a23-80cd-ba28e98da140"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:00:16.351628 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:16.351593    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbb8248f-a7c8-4a23-80cd-ba28e98da140-kube-api-access-mtrtz" pod "dbb8248f-a7c8-4a23-80cd-ba28e98da140" (UID: "dbb8248f-a7c8-4a23-80cd-ba28e98da140"). InnerVolumeSpecName "kube-api-access-mtrtz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:00:16.444166 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:16.443028    4477 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dbb8248f-a7c8-4a23-80cd-ba28e98da140-metrics-certs\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:00:16.444166 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:16.443084    4477 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbb8248f-a7c8-4a23-80cd-ba28e98da140-openshift-service-ca\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:00:16.444166 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:16.443099    4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtrtz\" (UniqueName: \"kubernetes.io/projected/dbb8248f-a7c8-4a23-80cd-ba28e98da140-kube-api-access-mtrtz\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:00:16.444166 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:16.443113    4477 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dbb8248f-a7c8-4a23-80cd-ba28e98da140-maas-api-tls\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:00:16.571090 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:16.568416    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e3a7dd-6b6b8b4cfd-9gg8g"
Oct 30 23:00:16.602322 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:16.602291    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e3a7dd-fc4fb5599-mg28w"
Oct 30 23:00:16.652462 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:16.652428    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04451835-ece6-4466-8b50-71aaf24b4828-kube-api-access-64wqd\" (UniqueName: \"kubernetes.io/projected/04451835-ece6-4466-8b50-71aaf24b4828-kube-api-access-64wqd\") pod \"04451835-ece6-4466-8b50-71aaf24b4828\" (UID: \"04451835-ece6-4466-8b50-71aaf24b4828\") "
Oct 30 23:00:16.652991 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:16.652491    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c178db77-b489-434c-99e3-c7934518a7f8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c178db77-b489-434c-99e3-c7934518a7f8-plugins-config-volume\") pod \"c178db77-b489-434c-99e3-c7934518a7f8\" (UID: \"c178db77-b489-434c-99e3-c7934518a7f8\") "
Oct 30 23:00:16.652991 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:16.652572    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04451835-ece6-4466-8b50-71aaf24b4828-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04451835-ece6-4466-8b50-71aaf24b4828-plugins-config-volume\") pod \"04451835-ece6-4466-8b50-71aaf24b4828\" (UID: \"04451835-ece6-4466-8b50-71aaf24b4828\") "
Oct 30 23:00:16.652991 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:16.652678    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c178db77-b489-434c-99e3-c7934518a7f8-kube-api-access-g5hjm\" (UniqueName: \"kubernetes.io/projected/c178db77-b489-434c-99e3-c7934518a7f8-kube-api-access-g5hjm\") pod \"c178db77-b489-434c-99e3-c7934518a7f8\" (UID: \"c178db77-b489-434c-99e3-c7934518a7f8\") "
Oct 30 23:00:16.652991 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:16.652961    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04451835-ece6-4466-8b50-71aaf24b4828-plugins-config-volume" pod "04451835-ece6-4466-8b50-71aaf24b4828" (UID: "04451835-ece6-4466-8b50-71aaf24b4828"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:00:16.653243 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:16.652966    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c178db77-b489-434c-99e3-c7934518a7f8-plugins-config-volume" pod "c178db77-b489-434c-99e3-c7934518a7f8" (UID: "c178db77-b489-434c-99e3-c7934518a7f8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:00:16.653355 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:16.653330    4477 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c178db77-b489-434c-99e3-c7934518a7f8-plugins-config-volume\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:00:16.653427 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:16.653353    4477 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04451835-ece6-4466-8b50-71aaf24b4828-plugins-config-volume\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:00:16.655642 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:16.655612    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04451835-ece6-4466-8b50-71aaf24b4828-kube-api-access-64wqd" pod "04451835-ece6-4466-8b50-71aaf24b4828" (UID: "04451835-ece6-4466-8b50-71aaf24b4828"). InnerVolumeSpecName "kube-api-access-64wqd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:00:16.656091 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:16.656052    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c178db77-b489-434c-99e3-c7934518a7f8-kube-api-access-g5hjm" pod "c178db77-b489-434c-99e3-c7934518a7f8" (UID: "c178db77-b489-434c-99e3-c7934518a7f8"). InnerVolumeSpecName "kube-api-access-g5hjm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:00:16.660154 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:16.660128    4477 generic.go:356] "Generic (PLEG): container finished" podID="dbb8248f-a7c8-4a23-80cd-ba28e98da140" containerID="272fddee0f30375f296ce1a617eb22a0752559e957bf069cf62e04e9ed249685" exitCode=0
Oct 30 23:00:16.660284 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:16.660169    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e3a7dd-866fc6f4cb-gqb2f"
Oct 30 23:00:16.660284 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:16.660211    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e3a7dd-866fc6f4cb-gqb2f" event={"ID":"dbb8248f-a7c8-4a23-80cd-ba28e98da140","Type":"ContainerDied","Data":"272fddee0f30375f296ce1a617eb22a0752559e957bf069cf62e04e9ed249685"}
Oct 30 23:00:16.660284 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:16.660261    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e3a7dd-866fc6f4cb-gqb2f" event={"ID":"dbb8248f-a7c8-4a23-80cd-ba28e98da140","Type":"ContainerDied","Data":"0d95721d320fd5170d74571d0879a891ac71c13bc5094231a118e1b906655554"}
Oct 30 23:00:16.660459 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:16.660291    4477 scope.go:122] "RemoveContainer" containerID="272fddee0f30375f296ce1a617eb22a0752559e957bf069cf62e04e9ed249685"
Oct 30 23:00:16.662698 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:16.662317    4477 generic.go:356] "Generic (PLEG): container finished" podID="04451835-ece6-4466-8b50-71aaf24b4828" containerID="0b8f3f1cdbda169c2a7c2ea51bdf5003cccf2428fc9b6df4473f8953414805b5" exitCode=0
Oct 30 23:00:16.662698 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:16.662393    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e3a7dd-6b6b8b4cfd-9gg8g" event={"ID":"04451835-ece6-4466-8b50-71aaf24b4828","Type":"ContainerDied","Data":"0b8f3f1cdbda169c2a7c2ea51bdf5003cccf2428fc9b6df4473f8953414805b5"}
Oct 30 23:00:16.662698 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:16.662437    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e3a7dd-6b6b8b4cfd-9gg8g" event={"ID":"04451835-ece6-4466-8b50-71aaf24b4828","Type":"ContainerDied","Data":"e8faf9a432ce87d7d4b08a94dd75b9ad093df86c81e4aa8c1cf914275b2a3d59"}
Oct 30 23:00:16.662698 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:16.662499    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e3a7dd-6b6b8b4cfd-9gg8g"
Oct 30 23:00:16.674620 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:16.670675    4477 generic.go:356] "Generic (PLEG): container finished" podID="c178db77-b489-434c-99e3-c7934518a7f8" containerID="43ef41e7bf29dfd1a3369830f718bbb11eedecd6381a8f283a099f8f5ab4ce3a" exitCode=0
Oct 30 23:00:16.674620 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:16.670846    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e3a7dd-fc4fb5599-mg28w" event={"ID":"c178db77-b489-434c-99e3-c7934518a7f8","Type":"ContainerDied","Data":"43ef41e7bf29dfd1a3369830f718bbb11eedecd6381a8f283a099f8f5ab4ce3a"}
Oct 30 23:00:16.674620 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:16.670925    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e3a7dd-fc4fb5599-mg28w" event={"ID":"c178db77-b489-434c-99e3-c7934518a7f8","Type":"ContainerDied","Data":"7144c86c880b7b4893317cd55f8947a348273559ce58bc708ab6cb7c5411ce95"}
Oct 30 23:00:16.674620 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:16.670998    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e3a7dd-fc4fb5599-mg28w"
Oct 30 23:00:16.702223 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:16.701206    4477 scope.go:122] "RemoveContainer" containerID="272fddee0f30375f296ce1a617eb22a0752559e957bf069cf62e04e9ed249685"
Oct 30 23:00:16.702223 ip-10-0-130-77 kubenswrapper[4477]: E1030 23:00:16.701588    4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"272fddee0f30375f296ce1a617eb22a0752559e957bf069cf62e04e9ed249685\": container with ID starting with 272fddee0f30375f296ce1a617eb22a0752559e957bf069cf62e04e9ed249685 not found: ID does not exist" containerID="272fddee0f30375f296ce1a617eb22a0752559e957bf069cf62e04e9ed249685"
Oct 30 23:00:16.702223 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:16.701621    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"272fddee0f30375f296ce1a617eb22a0752559e957bf069cf62e04e9ed249685"} err="failed to get container status \"272fddee0f30375f296ce1a617eb22a0752559e957bf069cf62e04e9ed249685\": rpc error: code = NotFound desc = could not find container \"272fddee0f30375f296ce1a617eb22a0752559e957bf069cf62e04e9ed249685\": container with ID starting with 272fddee0f30375f296ce1a617eb22a0752559e957bf069cf62e04e9ed249685 not found: ID does not exist"
Oct 30 23:00:16.702223 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:16.701639    4477 scope.go:122] "RemoveContainer" containerID="0b8f3f1cdbda169c2a7c2ea51bdf5003cccf2428fc9b6df4473f8953414805b5"
Oct 30 23:00:16.715969 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:16.715943    4477 scope.go:122] "RemoveContainer" containerID="0b8f3f1cdbda169c2a7c2ea51bdf5003cccf2428fc9b6df4473f8953414805b5"
Oct 30 23:00:16.716351 ip-10-0-130-77 kubenswrapper[4477]: E1030 23:00:16.716327    4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b8f3f1cdbda169c2a7c2ea51bdf5003cccf2428fc9b6df4473f8953414805b5\": container with ID starting with 0b8f3f1cdbda169c2a7c2ea51bdf5003cccf2428fc9b6df4473f8953414805b5 not found: ID does not exist" containerID="0b8f3f1cdbda169c2a7c2ea51bdf5003cccf2428fc9b6df4473f8953414805b5"
Oct 30 23:00:16.716458 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:16.716382    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b8f3f1cdbda169c2a7c2ea51bdf5003cccf2428fc9b6df4473f8953414805b5"} err="failed to get container status \"0b8f3f1cdbda169c2a7c2ea51bdf5003cccf2428fc9b6df4473f8953414805b5\": rpc error: code = NotFound desc = could not find container \"0b8f3f1cdbda169c2a7c2ea51bdf5003cccf2428fc9b6df4473f8953414805b5\": container with ID starting with 0b8f3f1cdbda169c2a7c2ea51bdf5003cccf2428fc9b6df4473f8953414805b5 not found: ID does not exist"
Oct 30 23:00:16.716458 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:16.716410    4477 scope.go:122] "RemoveContainer" containerID="43ef41e7bf29dfd1a3369830f718bbb11eedecd6381a8f283a099f8f5ab4ce3a"
Oct 30 23:00:16.730184 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:16.730164    4477 scope.go:122] "RemoveContainer" containerID="43ef41e7bf29dfd1a3369830f718bbb11eedecd6381a8f283a099f8f5ab4ce3a"
Oct 30 23:00:16.730453 ip-10-0-130-77 kubenswrapper[4477]: E1030 23:00:16.730427    4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43ef41e7bf29dfd1a3369830f718bbb11eedecd6381a8f283a099f8f5ab4ce3a\": container with ID starting with 43ef41e7bf29dfd1a3369830f718bbb11eedecd6381a8f283a099f8f5ab4ce3a not found: ID does not exist" containerID="43ef41e7bf29dfd1a3369830f718bbb11eedecd6381a8f283a099f8f5ab4ce3a"
Oct 30 23:00:16.730519 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:16.730460    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43ef41e7bf29dfd1a3369830f718bbb11eedecd6381a8f283a099f8f5ab4ce3a"} err="failed to get container status \"43ef41e7bf29dfd1a3369830f718bbb11eedecd6381a8f283a099f8f5ab4ce3a\": rpc error: code = NotFound desc = could not find container \"43ef41e7bf29dfd1a3369830f718bbb11eedecd6381a8f283a099f8f5ab4ce3a\": container with ID starting with 43ef41e7bf29dfd1a3369830f718bbb11eedecd6381a8f283a099f8f5ab4ce3a not found: ID does not exist"
Oct 30 23:00:16.754391 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:16.754362    4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64wqd\" (UniqueName: \"kubernetes.io/projected/04451835-ece6-4466-8b50-71aaf24b4828-kube-api-access-64wqd\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:00:16.754518 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:16.754394    4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5hjm\" (UniqueName: \"kubernetes.io/projected/c178db77-b489-434c-99e3-c7934518a7f8-kube-api-access-g5hjm\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:00:16.759363 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:16.759341    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e3a7dd-866fc6f4cb-gqb2f"]
Oct 30 23:00:16.769916 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:16.765477    4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e3a7dd-866fc6f4cb-gqb2f"]
Oct 30 23:00:16.861479 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:16.860208    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e3a7dd-fc4fb5599-mg28w"]
Oct 30 23:00:16.866496 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:16.863410    4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e3a7dd-fc4fb5599-mg28w"]
Oct 30 23:00:16.872415 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:16.872381    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-76d54a-29890020-dv4dj"
Oct 30 23:00:16.898880 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:16.898855    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e3a7dd-6b6b8b4cfd-9gg8g"]
Oct 30 23:00:16.904974 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:16.904948    4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e3a7dd-6b6b8b4cfd-9gg8g"]
Oct 30 23:00:16.962456 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:16.962383    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a910f2eb-f5b8-45fc-a827-103d116454f9-kube-api-access-lkfz8\" (UniqueName: \"kubernetes.io/projected/a910f2eb-f5b8-45fc-a827-103d116454f9-kube-api-access-lkfz8\") pod \"a910f2eb-f5b8-45fc-a827-103d116454f9\" (UID: \"a910f2eb-f5b8-45fc-a827-103d116454f9\") "
Oct 30 23:00:16.962594 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:16.962539    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a910f2eb-f5b8-45fc-a827-103d116454f9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a910f2eb-f5b8-45fc-a827-103d116454f9-maas-api-service-ca\") pod \"a910f2eb-f5b8-45fc-a827-103d116454f9\" (UID: \"a910f2eb-f5b8-45fc-a827-103d116454f9\") "
Oct 30 23:00:16.964001 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:16.963682    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a910f2eb-f5b8-45fc-a827-103d116454f9-maas-api-service-ca" pod "a910f2eb-f5b8-45fc-a827-103d116454f9" (UID: "a910f2eb-f5b8-45fc-a827-103d116454f9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:00:16.973567 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:16.973534    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a910f2eb-f5b8-45fc-a827-103d116454f9-kube-api-access-lkfz8" pod "a910f2eb-f5b8-45fc-a827-103d116454f9" (UID: "a910f2eb-f5b8-45fc-a827-103d116454f9"). InnerVolumeSpecName "kube-api-access-lkfz8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:00:17.064378 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:17.064353    4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkfz8\" (UniqueName: \"kubernetes.io/projected/a910f2eb-f5b8-45fc-a827-103d116454f9-kube-api-access-lkfz8\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:00:17.064378 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:17.064377    4477 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a910f2eb-f5b8-45fc-a827-103d116454f9-maas-api-service-ca\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:00:17.186776 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:17.186750    4477 scope.go:122] "RemoveContainer" containerID="e4222498307eeb32298453f7fd68480abf6e9fb3f7a5162d509cf379b02b7ed3"
Oct 30 23:00:17.695076 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:17.695029    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-76d54a-29890020-dv4dj"
Oct 30 23:00:17.695572 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:17.695085    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-76d54a-29890020-dv4dj" event={"ID":"a910f2eb-f5b8-45fc-a827-103d116454f9","Type":"ContainerDied","Data":"b74cf2ff553441bc04ebc9be491dab5beed7466f062fc0c06e25dd25adf1607e"}
Oct 30 23:00:17.695572 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:17.695158    4477 scope.go:122] "RemoveContainer" containerID="64ec9470be9a199c70d84ae81672637b4d89f0ffe935df38e8f2e983b43359c2"
Oct 30 23:00:17.697163 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:17.697134    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f88979-29890020-ngbtw" event={"ID":"cdcfe2c6-edf0-42a3-ac07-434812bedaa5","Type":"ContainerStarted","Data":"430f37f759251289630f32401a4143beec57fae488128ac6a9ce0989af2acaca"}
Oct 30 23:00:17.745429 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:17.745378    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-76d54a-29890020-dv4dj"]
Oct 30 23:00:17.751071 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:17.751024    4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-76d54a-29890020-dv4dj"]
Oct 30 23:00:18.193868 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:18.193841    4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04451835-ece6-4466-8b50-71aaf24b4828" path="/var/lib/kubelet/pods/04451835-ece6-4466-8b50-71aaf24b4828/volumes"
Oct 30 23:00:18.194288 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:18.194273    4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a910f2eb-f5b8-45fc-a827-103d116454f9" path="/var/lib/kubelet/pods/a910f2eb-f5b8-45fc-a827-103d116454f9/volumes"
Oct 30 23:00:18.194657 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:18.194645    4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c178db77-b489-434c-99e3-c7934518a7f8" path="/var/lib/kubelet/pods/c178db77-b489-434c-99e3-c7934518a7f8/volumes"
Oct 30 23:00:18.195151 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:18.195131    4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbb8248f-a7c8-4a23-80cd-ba28e98da140" path="/var/lib/kubelet/pods/dbb8248f-a7c8-4a23-80cd-ba28e98da140/volumes"
Oct 30 23:00:18.706578 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:18.706545    4477 generic.go:356] "Generic (PLEG): container finished" podID="cdcfe2c6-edf0-42a3-ac07-434812bedaa5" containerID="430f37f759251289630f32401a4143beec57fae488128ac6a9ce0989af2acaca" exitCode=60
Oct 30 23:00:18.706977 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:18.706620    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f88979-29890020-ngbtw" event={"ID":"cdcfe2c6-edf0-42a3-ac07-434812bedaa5","Type":"ContainerDied","Data":"430f37f759251289630f32401a4143beec57fae488128ac6a9ce0989af2acaca"}
Oct 30 23:00:18.706977 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:18.706729    4477 scope.go:122] "RemoveContainer" containerID="e4222498307eeb32298453f7fd68480abf6e9fb3f7a5162d509cf379b02b7ed3"
Oct 30 23:00:18.707180 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:18.707167    4477 scope.go:122] "RemoveContainer" containerID="430f37f759251289630f32401a4143beec57fae488128ac6a9ce0989af2acaca"
Oct 30 23:00:18.707481 ip-10-0-130-77 kubenswrapper[4477]: E1030 23:00:18.707461    4477 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-f88979-29890020-ngbtw_odh-ai-gateway-infra(cdcfe2c6-edf0-42a3-ac07-434812bedaa5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f88979-29890020-ngbtw" podUID="cdcfe2c6-edf0-42a3-ac07-434812bedaa5"
Oct 30 23:00:18.803371 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:18.803333    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f88979-29890020-ngbtw"]
Oct 30 23:00:19.864274 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:19.864247    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f88979-29890020-ngbtw"
Oct 30 23:00:19.907126 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:19.907098    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdcfe2c6-edf0-42a3-ac07-434812bedaa5-kube-api-access-jqvfr\" (UniqueName: \"kubernetes.io/projected/cdcfe2c6-edf0-42a3-ac07-434812bedaa5-kube-api-access-jqvfr\") pod \"cdcfe2c6-edf0-42a3-ac07-434812bedaa5\" (UID: \"cdcfe2c6-edf0-42a3-ac07-434812bedaa5\") "
Oct 30 23:00:19.907280 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:19.907261    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cdcfe2c6-edf0-42a3-ac07-434812bedaa5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdcfe2c6-edf0-42a3-ac07-434812bedaa5-maas-api-service-ca\") pod \"cdcfe2c6-edf0-42a3-ac07-434812bedaa5\" (UID: \"cdcfe2c6-edf0-42a3-ac07-434812bedaa5\") "
Oct 30 23:00:19.907964 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:19.907936    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdcfe2c6-edf0-42a3-ac07-434812bedaa5-maas-api-service-ca" pod "cdcfe2c6-edf0-42a3-ac07-434812bedaa5" (UID: "cdcfe2c6-edf0-42a3-ac07-434812bedaa5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:00:19.909656 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:19.909633    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdcfe2c6-edf0-42a3-ac07-434812bedaa5-kube-api-access-jqvfr" pod "cdcfe2c6-edf0-42a3-ac07-434812bedaa5" (UID: "cdcfe2c6-edf0-42a3-ac07-434812bedaa5"). InnerVolumeSpecName "kube-api-access-jqvfr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:00:20.009391 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:20.009305    4477 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cdcfe2c6-edf0-42a3-ac07-434812bedaa5-maas-api-service-ca\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:00:20.009391 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:20.009363    4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqvfr\" (UniqueName: \"kubernetes.io/projected/cdcfe2c6-edf0-42a3-ac07-434812bedaa5-kube-api-access-jqvfr\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:00:20.723525 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:20.723493    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f88979-29890020-ngbtw"
Oct 30 23:00:20.723729 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:20.723491    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f88979-29890020-ngbtw" event={"ID":"cdcfe2c6-edf0-42a3-ac07-434812bedaa5","Type":"ContainerDied","Data":"a315f72dcd6eeb414e748735bd0e847634de786f7943db45f0b927587f26cd94"}
Oct 30 23:00:20.723729 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:20.723645    4477 scope.go:122] "RemoveContainer" containerID="430f37f759251289630f32401a4143beec57fae488128ac6a9ce0989af2acaca"
Oct 30 23:00:20.786087 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:20.786056    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f88979-29890020-ngbtw"]
Oct 30 23:00:20.797294 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:20.797267    4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-f88979-29890020-ngbtw"]
Oct 30 23:00:22.194328 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:22.194292    4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cdcfe2c6-edf0-42a3-ac07-434812bedaa5" path="/var/lib/kubelet/pods/cdcfe2c6-edf0-42a3-ac07-434812bedaa5/volumes"
Oct 30 23:00:26.266663 ip-10-0-130-77 kubenswrapper[4477]: E1030 23:00:26.266627    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 23:00:43.265339 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:00:43.265303    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6bz92"]
Oct 30 23:00:56.267083 ip-10-0-130-77 kubenswrapper[4477]: E1030 23:00:56.267024    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 23:01:26.266170 ip-10-0-130-77 kubenswrapper[4477]: E1030 23:01:26.266134    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 23:01:48.782548 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:01:48.782510    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6bz92"]
Oct 30 23:01:55.812969 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:01:55.812935    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a60e80-6fdbc5f7f9-24lzn"]
Oct 30 23:01:55.813524 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:01:55.813151    4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a60e80-6fdbc5f7f9-24lzn" podUID="d034fb1b-68f3-437a-ba54-f4537069c282" containerName="maas-api" containerID="cri-o://4913e6b3dfc0b44fa71df9de18b9027df075d7d7f6a1e72ac7cc806cbb80d7c9" gracePeriod=30
Oct 30 23:01:56.118559 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:01:56.118531    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a60e80-6fdbc5f7f9-24lzn"
Oct 30 23:01:56.221654 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:01:56.221623    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d034fb1b-68f3-437a-ba54-f4537069c282-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d034fb1b-68f3-437a-ba54-f4537069c282-maas-api-tls\") pod \"d034fb1b-68f3-437a-ba54-f4537069c282\" (UID: \"d034fb1b-68f3-437a-ba54-f4537069c282\") "
Oct 30 23:01:56.221827 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:01:56.221713    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d034fb1b-68f3-437a-ba54-f4537069c282-kube-api-access-jsvxp\" (UniqueName: \"kubernetes.io/projected/d034fb1b-68f3-437a-ba54-f4537069c282-kube-api-access-jsvxp\") pod \"d034fb1b-68f3-437a-ba54-f4537069c282\" (UID: \"d034fb1b-68f3-437a-ba54-f4537069c282\") "
Oct 30 23:01:56.221907 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:01:56.221879    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d034fb1b-68f3-437a-ba54-f4537069c282-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d034fb1b-68f3-437a-ba54-f4537069c282-metrics-certs\") pod \"d034fb1b-68f3-437a-ba54-f4537069c282\" (UID: \"d034fb1b-68f3-437a-ba54-f4537069c282\") "
Oct 30 23:01:56.221967 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:01:56.221924    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d034fb1b-68f3-437a-ba54-f4537069c282-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d034fb1b-68f3-437a-ba54-f4537069c282-openshift-service-ca\") pod \"d034fb1b-68f3-437a-ba54-f4537069c282\" (UID: \"d034fb1b-68f3-437a-ba54-f4537069c282\") "
Oct 30 23:01:56.222735 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:01:56.222697    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d034fb1b-68f3-437a-ba54-f4537069c282-openshift-service-ca" pod "d034fb1b-68f3-437a-ba54-f4537069c282" (UID: "d034fb1b-68f3-437a-ba54-f4537069c282"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:01:56.223850 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:01:56.223821    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d034fb1b-68f3-437a-ba54-f4537069c282-kube-api-access-jsvxp" pod "d034fb1b-68f3-437a-ba54-f4537069c282" (UID: "d034fb1b-68f3-437a-ba54-f4537069c282"). InnerVolumeSpecName "kube-api-access-jsvxp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:01:56.223968 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:01:56.223843    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d034fb1b-68f3-437a-ba54-f4537069c282-maas-api-tls" pod "d034fb1b-68f3-437a-ba54-f4537069c282" (UID: "d034fb1b-68f3-437a-ba54-f4537069c282"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:01:56.223968 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:01:56.223923    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d034fb1b-68f3-437a-ba54-f4537069c282-metrics-certs" pod "d034fb1b-68f3-437a-ba54-f4537069c282" (UID: "d034fb1b-68f3-437a-ba54-f4537069c282"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:01:56.270607 ip-10-0-130-77 kubenswrapper[4477]: E1030 23:01:56.270543    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 23:01:56.345115 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:01:56.343257    4477 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d034fb1b-68f3-437a-ba54-f4537069c282-maas-api-tls\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:01:56.345115 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:01:56.343296    4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jsvxp\" (UniqueName: \"kubernetes.io/projected/d034fb1b-68f3-437a-ba54-f4537069c282-kube-api-access-jsvxp\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:01:56.345115 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:01:56.343314    4477 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d034fb1b-68f3-437a-ba54-f4537069c282-metrics-certs\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:01:56.345115 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:01:56.343329    4477 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d034fb1b-68f3-437a-ba54-f4537069c282-openshift-service-ca\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:01:56.379495 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:01:56.379455    4477 generic.go:356] "Generic (PLEG): container finished" podID="d034fb1b-68f3-437a-ba54-f4537069c282" containerID="4913e6b3dfc0b44fa71df9de18b9027df075d7d7f6a1e72ac7cc806cbb80d7c9" exitCode=0
Oct 30 23:01:56.379690 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:01:56.379508    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a60e80-6fdbc5f7f9-24lzn"
Oct 30 23:01:56.379690 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:01:56.379539    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a60e80-6fdbc5f7f9-24lzn" event={"ID":"d034fb1b-68f3-437a-ba54-f4537069c282","Type":"ContainerDied","Data":"4913e6b3dfc0b44fa71df9de18b9027df075d7d7f6a1e72ac7cc806cbb80d7c9"}
Oct 30 23:01:56.379690 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:01:56.379600    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-a60e80-6fdbc5f7f9-24lzn" event={"ID":"d034fb1b-68f3-437a-ba54-f4537069c282","Type":"ContainerDied","Data":"6677bcdfb3dc7eac69131c418e73877de2a4ba0715f8bb84f678b17d67e37c3c"}
Oct 30 23:01:56.379690 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:01:56.379622    4477 scope.go:122] "RemoveContainer" containerID="4913e6b3dfc0b44fa71df9de18b9027df075d7d7f6a1e72ac7cc806cbb80d7c9"
Oct 30 23:01:56.395181 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:01:56.395159    4477 scope.go:122] "RemoveContainer" containerID="4913e6b3dfc0b44fa71df9de18b9027df075d7d7f6a1e72ac7cc806cbb80d7c9"
Oct 30 23:01:56.395480 ip-10-0-130-77 kubenswrapper[4477]: E1030 23:01:56.395459    4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4913e6b3dfc0b44fa71df9de18b9027df075d7d7f6a1e72ac7cc806cbb80d7c9\": container with ID starting with 4913e6b3dfc0b44fa71df9de18b9027df075d7d7f6a1e72ac7cc806cbb80d7c9 not found: ID does not exist" containerID="4913e6b3dfc0b44fa71df9de18b9027df075d7d7f6a1e72ac7cc806cbb80d7c9"
Oct 30 23:01:56.395549 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:01:56.395490    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4913e6b3dfc0b44fa71df9de18b9027df075d7d7f6a1e72ac7cc806cbb80d7c9"} err="failed to get container status \"4913e6b3dfc0b44fa71df9de18b9027df075d7d7f6a1e72ac7cc806cbb80d7c9\": rpc error: code = NotFound desc = could not find container \"4913e6b3dfc0b44fa71df9de18b9027df075d7d7f6a1e72ac7cc806cbb80d7c9\": container with ID starting with 4913e6b3dfc0b44fa71df9de18b9027df075d7d7f6a1e72ac7cc806cbb80d7c9 not found: ID does not exist"
Oct 30 23:01:56.427554 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:01:56.427529    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a60e80-6fdbc5f7f9-24lzn"]
Oct 30 23:01:56.432747 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:01:56.432726    4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-a60e80-6fdbc5f7f9-24lzn"]
Oct 30 23:01:57.118082 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:01:57.117951    4477 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jcf8l_015f218d-bf1f-4e7e-bb5d-b805af003246/ovn-acl-logging/0.log"
Oct 30 23:01:57.130129 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:01:57.127700    4477 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jcf8l_015f218d-bf1f-4e7e-bb5d-b805af003246/ovn-acl-logging/0.log"
Oct 30 23:01:58.194204 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:01:58.194172    4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d034fb1b-68f3-437a-ba54-f4537069c282" path="/var/lib/kubelet/pods/d034fb1b-68f3-437a-ba54-f4537069c282/volumes"
Oct 30 23:02:26.266378 ip-10-0-130-77 kubenswrapper[4477]: E1030 23:02:26.266335    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 23:02:28.226314 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:02:28.226280    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-bff950-c46cf8dc-95t2k"]
Oct 30 23:02:28.226851 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:02:28.226833    4477 cpu_manager.go:414] "containerMap: removing container" podUID="a910f2eb-f5b8-45fc-a827-103d116454f9" containerName="cleanup"
Oct 30 23:02:28.226851 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:02:28.226851    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a910f2eb-f5b8-45fc-a827-103d116454f9" containerName="cleanup"
Oct 30 23:02:28.226970 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:02:28.226871    4477 cpu_manager.go:414] "containerMap: removing container" podUID="dbb8248f-a7c8-4a23-80cd-ba28e98da140" containerName="maas-api"
Oct 30 23:02:28.226970 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:02:28.226876    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbb8248f-a7c8-4a23-80cd-ba28e98da140" containerName="maas-api"
Oct 30 23:02:28.226970 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:02:28.226885    4477 cpu_manager.go:414] "containerMap: removing container" podUID="d034fb1b-68f3-437a-ba54-f4537069c282" containerName="maas-api"
Oct 30 23:02:28.226970 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:02:28.226890    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d034fb1b-68f3-437a-ba54-f4537069c282" containerName="maas-api"
Oct 30 23:02:28.226970 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:02:28.226900    4477 cpu_manager.go:414] "containerMap: removing container" podUID="c178db77-b489-434c-99e3-c7934518a7f8" containerName="payload-pre-processing"
Oct 30 23:02:28.226970 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:02:28.226905    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c178db77-b489-434c-99e3-c7934518a7f8" containerName="payload-pre-processing"
Oct 30 23:02:28.226970 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:02:28.226913    4477 cpu_manager.go:414] "containerMap: removing container" podUID="cdcfe2c6-edf0-42a3-ac07-434812bedaa5" containerName="cleanup"
Oct 30 23:02:28.226970 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:02:28.226922    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdcfe2c6-edf0-42a3-ac07-434812bedaa5" containerName="cleanup"
Oct 30 23:02:28.226970 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:02:28.226953    4477 cpu_manager.go:414] "containerMap: removing container" podUID="a910f2eb-f5b8-45fc-a827-103d116454f9" containerName="cleanup"
Oct 30 23:02:28.226970 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:02:28.226957    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a910f2eb-f5b8-45fc-a827-103d116454f9" containerName="cleanup"
Oct 30 23:02:28.226970 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:02:28.226964    4477 cpu_manager.go:414] "containerMap: removing container" podUID="cdcfe2c6-edf0-42a3-ac07-434812bedaa5" containerName="cleanup"
Oct 30 23:02:28.226970 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:02:28.226969    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdcfe2c6-edf0-42a3-ac07-434812bedaa5" containerName="cleanup"
Oct 30 23:02:28.226970 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:02:28.226976    4477 cpu_manager.go:414] "containerMap: removing container" podUID="04451835-ece6-4466-8b50-71aaf24b4828" containerName="payload-processing"
Oct 30 23:02:28.227616 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:02:28.226981    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04451835-ece6-4466-8b50-71aaf24b4828" containerName="payload-processing"
Oct 30 23:02:28.227616 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:02:28.227110    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="a910f2eb-f5b8-45fc-a827-103d116454f9" containerName="cleanup"
Oct 30 23:02:28.227616 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:02:28.227118    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdcfe2c6-edf0-42a3-ac07-434812bedaa5" containerName="cleanup"
Oct 30 23:02:28.227616 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:02:28.227124    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="04451835-ece6-4466-8b50-71aaf24b4828" containerName="payload-processing"
Oct 30 23:02:28.227616 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:02:28.227132    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="a910f2eb-f5b8-45fc-a827-103d116454f9" containerName="cleanup"
Oct 30 23:02:28.227616 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:02:28.227142    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbb8248f-a7c8-4a23-80cd-ba28e98da140" containerName="maas-api"
Oct 30 23:02:28.227616 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:02:28.227150    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="d034fb1b-68f3-437a-ba54-f4537069c282" containerName="maas-api"
Oct 30 23:02:28.227616 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:02:28.227156    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="c178db77-b489-434c-99e3-c7934518a7f8" containerName="payload-pre-processing"
Oct 30 23:02:28.227616 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:02:28.227166    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdcfe2c6-edf0-42a3-ac07-434812bedaa5" containerName="cleanup"
Oct 30 23:02:28.230789 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:02:28.230765    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bff950-c46cf8dc-95t2k"
Oct 30 23:02:28.254737 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:02:28.254713    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-bff950-serving-cert\""
Oct 30 23:02:28.257586 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:02:28.257563    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-bff950-c46cf8dc-95t2k"]
Oct 30 23:02:28.346573 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:02:28.346543    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a637e6be-489f-4d98-bc02-bca751282d13-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-bff950-c46cf8dc-95t2k\" (UID: \"a637e6be-489f-4d98-bc02-bca751282d13\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bff950-c46cf8dc-95t2k"
Oct 30 23:02:28.346722 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:02:28.346587    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a637e6be-489f-4d98-bc02-bca751282d13-maas-api-tls\") pod \"maas-api-e2e-ipp-b-bff950-c46cf8dc-95t2k\" (UID: \"a637e6be-489f-4d98-bc02-bca751282d13\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bff950-c46cf8dc-95t2k"
Oct 30 23:02:28.346793 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:02:28.346775    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hvf2\" (UniqueName: \"kubernetes.io/projected/a637e6be-489f-4d98-bc02-bca751282d13-kube-api-access-6hvf2\") pod \"maas-api-e2e-ipp-b-bff950-c46cf8dc-95t2k\" (UID: \"a637e6be-489f-4d98-bc02-bca751282d13\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bff950-c46cf8dc-95t2k"
Oct 30 23:02:28.346832 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:02:28.346823    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a637e6be-489f-4d98-bc02-bca751282d13-metrics-certs\") pod \"maas-api-e2e-ipp-b-bff950-c46cf8dc-95t2k\" (UID: \"a637e6be-489f-4d98-bc02-bca751282d13\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bff950-c46cf8dc-95t2k"
Oct 30 23:02:28.464478 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:02:28.452734    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a637e6be-489f-4d98-bc02-bca751282d13-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-bff950-c46cf8dc-95t2k\" (UID: \"a637e6be-489f-4d98-bc02-bca751282d13\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bff950-c46cf8dc-95t2k"
Oct 30 23:02:28.470101 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:02:28.468557    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a637e6be-489f-4d98-bc02-bca751282d13-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-bff950-c46cf8dc-95t2k\" (UID: \"a637e6be-489f-4d98-bc02-bca751282d13\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bff950-c46cf8dc-95t2k"
Oct 30 23:02:28.470387 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:02:28.470360    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a637e6be-489f-4d98-bc02-bca751282d13-maas-api-tls\") pod \"maas-api-e2e-ipp-b-bff950-c46cf8dc-95t2k\" (UID: \"a637e6be-489f-4d98-bc02-bca751282d13\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bff950-c46cf8dc-95t2k"
Oct 30 23:02:28.470744 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:02:28.470716    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hvf2\" (UniqueName: \"kubernetes.io/projected/a637e6be-489f-4d98-bc02-bca751282d13-kube-api-access-6hvf2\") pod \"maas-api-e2e-ipp-b-bff950-c46cf8dc-95t2k\" (UID: \"a637e6be-489f-4d98-bc02-bca751282d13\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bff950-c46cf8dc-95t2k"
Oct 30 23:02:28.473887 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:02:28.470916    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a637e6be-489f-4d98-bc02-bca751282d13-metrics-certs\") pod \"maas-api-e2e-ipp-b-bff950-c46cf8dc-95t2k\" (UID: \"a637e6be-489f-4d98-bc02-bca751282d13\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bff950-c46cf8dc-95t2k"
Oct 30 23:02:28.498896 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:02:28.498478    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a637e6be-489f-4d98-bc02-bca751282d13-metrics-certs\") pod \"maas-api-e2e-ipp-b-bff950-c46cf8dc-95t2k\" (UID: \"a637e6be-489f-4d98-bc02-bca751282d13\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bff950-c46cf8dc-95t2k"
Oct 30 23:02:28.504554 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:02:28.504444    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hvf2\" (UniqueName: \"kubernetes.io/projected/a637e6be-489f-4d98-bc02-bca751282d13-kube-api-access-6hvf2\") pod \"maas-api-e2e-ipp-b-bff950-c46cf8dc-95t2k\" (UID: \"a637e6be-489f-4d98-bc02-bca751282d13\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bff950-c46cf8dc-95t2k"
Oct 30 23:02:28.515097 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:02:28.514999    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a637e6be-489f-4d98-bc02-bca751282d13-maas-api-tls\") pod \"maas-api-e2e-ipp-b-bff950-c46cf8dc-95t2k\" (UID: \"a637e6be-489f-4d98-bc02-bca751282d13\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bff950-c46cf8dc-95t2k"
Oct 30 23:02:28.546655 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:02:28.546627    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bff950-c46cf8dc-95t2k"
Oct 30 23:02:28.849777 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:02:28.849698    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-bff950-c46cf8dc-95t2k"]
Oct 30 23:02:28.854034 ip-10-0-130-77 kubenswrapper[4477]: W1030 23:02:28.853991    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda637e6be_489f_4d98_bc02_bca751282d13.slice/crio-fa88e20845a2469fdadc9aa042dca866e725c4f35d2f3538b0d2551c1191e0fa WatchSource:0}: Error finding container fa88e20845a2469fdadc9aa042dca866e725c4f35d2f3538b0d2551c1191e0fa: Status 404 returned error can't find the container with id fa88e20845a2469fdadc9aa042dca866e725c4f35d2f3538b0d2551c1191e0fa
Oct 30 23:02:29.590636 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:02:29.590548    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bff950-c46cf8dc-95t2k" event={"ID":"a637e6be-489f-4d98-bc02-bca751282d13","Type":"ContainerStarted","Data":"54194ed0b25ccae4f9d3a678bf02c6b0899ffb8fa8998b98ecf81b7f0d999560"}
Oct 30 23:02:29.590636 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:02:29.590613    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bff950-c46cf8dc-95t2k" event={"ID":"a637e6be-489f-4d98-bc02-bca751282d13","Type":"ContainerStarted","Data":"fa88e20845a2469fdadc9aa042dca866e725c4f35d2f3538b0d2551c1191e0fa"}
Oct 30 23:02:29.591110 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:02:29.590642    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bff950-c46cf8dc-95t2k"
Oct 30 23:02:29.629786 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:02:29.629725    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bff950-c46cf8dc-95t2k" podStartSLOduration=1.220092054 podStartE2EDuration="1.629706102s" totalImagesPullingTime="409.614048ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:02:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:02:29.626870685 +0000 UTC m=+3333.552796891" watchObservedRunningTime="2026-10-30 23:02:29.629706102 +0000 UTC m=+3333.555632308"
Oct 30 23:02:34.245530 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:02:34.245495    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-6f2eba-787bf86545-5pm9l"]
Oct 30 23:02:34.246088 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:02:34.245723    4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-6f2eba-787bf86545-5pm9l" podUID="2d8ff9bb-aa7c-46f4-8ff8-d85ba96870a4" containerName="maas-api" containerID="cri-o://d508ab4a260bbe48ff27d505bdba7309118828242a01bed72c6b54aad6f84376" gracePeriod=30
Oct 30 23:02:34.624152 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:02:34.624112    4477 generic.go:356] "Generic (PLEG): container finished" podID="2d8ff9bb-aa7c-46f4-8ff8-d85ba96870a4" containerID="d508ab4a260bbe48ff27d505bdba7309118828242a01bed72c6b54aad6f84376" exitCode=0
Oct 30 23:02:34.624325 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:02:34.624198    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-6f2eba-787bf86545-5pm9l" event={"ID":"2d8ff9bb-aa7c-46f4-8ff8-d85ba96870a4","Type":"ContainerDied","Data":"d508ab4a260bbe48ff27d505bdba7309118828242a01bed72c6b54aad6f84376"}
Oct 30 23:02:34.624325 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:02:34.624248    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-6f2eba-787bf86545-5pm9l" event={"ID":"2d8ff9bb-aa7c-46f4-8ff8-d85ba96870a4","Type":"ContainerDied","Data":"69e7e31cbd87bbb6cd131e599708fdd8c4dce153a2da67db9542706080b4e591"}
Oct 30 23:02:34.624325 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:02:34.624273    4477 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="69e7e31cbd87bbb6cd131e599708fdd8c4dce153a2da67db9542706080b4e591"
Oct 30 23:02:34.639214 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:02:34.639188    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-6f2eba-787bf86545-5pm9l"
Oct 30 23:02:34.659821 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:02:34.659792    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d8ff9bb-aa7c-46f4-8ff8-d85ba96870a4-kube-api-access-cclv7\" (UniqueName: \"kubernetes.io/projected/2d8ff9bb-aa7c-46f4-8ff8-d85ba96870a4-kube-api-access-cclv7\") pod \"2d8ff9bb-aa7c-46f4-8ff8-d85ba96870a4\" (UID: \"2d8ff9bb-aa7c-46f4-8ff8-d85ba96870a4\") "
Oct 30 23:02:34.659941 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:02:34.659898    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d8ff9bb-aa7c-46f4-8ff8-d85ba96870a4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d8ff9bb-aa7c-46f4-8ff8-d85ba96870a4-maas-api-tls\") pod \"2d8ff9bb-aa7c-46f4-8ff8-d85ba96870a4\" (UID: \"2d8ff9bb-aa7c-46f4-8ff8-d85ba96870a4\") "
Oct 30 23:02:34.660005 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:02:34.659956    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d8ff9bb-aa7c-46f4-8ff8-d85ba96870a4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d8ff9bb-aa7c-46f4-8ff8-d85ba96870a4-openshift-service-ca\") pod \"2d8ff9bb-aa7c-46f4-8ff8-d85ba96870a4\" (UID: \"2d8ff9bb-aa7c-46f4-8ff8-d85ba96870a4\") "
Oct 30 23:02:34.660412 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:02:34.660385    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d8ff9bb-aa7c-46f4-8ff8-d85ba96870a4-openshift-service-ca" pod "2d8ff9bb-aa7c-46f4-8ff8-d85ba96870a4" (UID: "2d8ff9bb-aa7c-46f4-8ff8-d85ba96870a4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:02:34.664878 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:02:34.662581    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d8ff9bb-aa7c-46f4-8ff8-d85ba96870a4-kube-api-access-cclv7" pod "2d8ff9bb-aa7c-46f4-8ff8-d85ba96870a4" (UID: "2d8ff9bb-aa7c-46f4-8ff8-d85ba96870a4"). InnerVolumeSpecName "kube-api-access-cclv7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:02:34.664878 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:02:34.662665    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d8ff9bb-aa7c-46f4-8ff8-d85ba96870a4-maas-api-tls" pod "2d8ff9bb-aa7c-46f4-8ff8-d85ba96870a4" (UID: "2d8ff9bb-aa7c-46f4-8ff8-d85ba96870a4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:02:34.761171 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:02:34.761120    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d8ff9bb-aa7c-46f4-8ff8-d85ba96870a4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d8ff9bb-aa7c-46f4-8ff8-d85ba96870a4-metrics-certs\") pod \"2d8ff9bb-aa7c-46f4-8ff8-d85ba96870a4\" (UID: \"2d8ff9bb-aa7c-46f4-8ff8-d85ba96870a4\") "
Oct 30 23:02:34.762084 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:02:34.762036    4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cclv7\" (UniqueName: \"kubernetes.io/projected/2d8ff9bb-aa7c-46f4-8ff8-d85ba96870a4-kube-api-access-cclv7\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:02:34.762260 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:02:34.762087    4477 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2d8ff9bb-aa7c-46f4-8ff8-d85ba96870a4-maas-api-tls\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:02:34.762260 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:02:34.762100    4477 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d8ff9bb-aa7c-46f4-8ff8-d85ba96870a4-openshift-service-ca\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:02:34.765123 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:02:34.765090    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d8ff9bb-aa7c-46f4-8ff8-d85ba96870a4-metrics-certs" pod "2d8ff9bb-aa7c-46f4-8ff8-d85ba96870a4" (UID: "2d8ff9bb-aa7c-46f4-8ff8-d85ba96870a4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:02:34.863414 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:02:34.863333    4477 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2d8ff9bb-aa7c-46f4-8ff8-d85ba96870a4-metrics-certs\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:02:35.602280 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:02:35.602210    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bff950-c46cf8dc-95t2k"
Oct 30 23:02:35.630061 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:02:35.630020    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-6f2eba-787bf86545-5pm9l"
Oct 30 23:02:35.693345 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:02:35.693311    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-6f2eba-787bf86545-5pm9l"]
Oct 30 23:02:35.712247 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:02:35.712136    4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-6f2eba-787bf86545-5pm9l"]
Oct 30 23:02:36.194181 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:02:36.194147    4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d8ff9bb-aa7c-46f4-8ff8-d85ba96870a4" path="/var/lib/kubelet/pods/2d8ff9bb-aa7c-46f4-8ff8-d85ba96870a4/volumes"
Oct 30 23:02:55.259439 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:02:55.259396    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6bz92"]
Oct 30 23:02:56.149988 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:02:56.149949    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-bmccc"]
Oct 30 23:02:56.150234 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:02:56.150195    4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-bmccc" podUID="30fed99f-b5e6-4c12-a1f4-8d84ed15971e" containerName="istio-proxy" containerID="cri-o://0781d508d5529cb9d8676b66eafa6284ea19d8b76451ac40ce1aac9910076060" gracePeriod=30
Oct 30 23:02:56.266943 ip-10-0-130-77 kubenswrapper[4477]: E1030 23:02:56.266906    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 23:02:57.426550 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:02:57.426519    4477 scope.go:122] "RemoveContainer" containerID="d508ab4a260bbe48ff27d505bdba7309118828242a01bed72c6b54aad6f84376"
Oct 30 23:03:01.418738 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:01.418682    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-bmccc"
Oct 30 23:03:01.515127 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:01.515100    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/30fed99f-b5e6-4c12-a1f4-8d84ed15971e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30fed99f-b5e6-4c12-a1f4-8d84ed15971e-istio-podinfo\") pod \"30fed99f-b5e6-4c12-a1f4-8d84ed15971e\" (UID: \"30fed99f-b5e6-4c12-a1f4-8d84ed15971e\") "
Oct 30 23:03:01.515281 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:01.515187    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30fed99f-b5e6-4c12-a1f4-8d84ed15971e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30fed99f-b5e6-4c12-a1f4-8d84ed15971e-istiod-ca-cert\") pod \"30fed99f-b5e6-4c12-a1f4-8d84ed15971e\" (UID: \"30fed99f-b5e6-4c12-a1f4-8d84ed15971e\") "
Oct 30 23:03:01.515281 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:01.515248    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30fed99f-b5e6-4c12-a1f4-8d84ed15971e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30fed99f-b5e6-4c12-a1f4-8d84ed15971e-istio-envoy\") pod \"30fed99f-b5e6-4c12-a1f4-8d84ed15971e\" (UID: \"30fed99f-b5e6-4c12-a1f4-8d84ed15971e\") "
Oct 30 23:03:01.515389 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:01.515282    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30fed99f-b5e6-4c12-a1f4-8d84ed15971e-kube-api-access-j6dhb\" (UniqueName: \"kubernetes.io/projected/30fed99f-b5e6-4c12-a1f4-8d84ed15971e-kube-api-access-j6dhb\") pod \"30fed99f-b5e6-4c12-a1f4-8d84ed15971e\" (UID: \"30fed99f-b5e6-4c12-a1f4-8d84ed15971e\") "
Oct 30 23:03:01.515443 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:01.515398    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30fed99f-b5e6-4c12-a1f4-8d84ed15971e-istio-token\" (UniqueName: \"kubernetes.io/projected/30fed99f-b5e6-4c12-a1f4-8d84ed15971e-istio-token\") pod \"30fed99f-b5e6-4c12-a1f4-8d84ed15971e\" (UID: \"30fed99f-b5e6-4c12-a1f4-8d84ed15971e\") "
Oct 30 23:03:01.515510 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:01.515449    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30fed99f-b5e6-4c12-a1f4-8d84ed15971e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30fed99f-b5e6-4c12-a1f4-8d84ed15971e-istio-data\") pod \"30fed99f-b5e6-4c12-a1f4-8d84ed15971e\" (UID: \"30fed99f-b5e6-4c12-a1f4-8d84ed15971e\") "
Oct 30 23:03:01.515510 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:01.515504    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30fed99f-b5e6-4c12-a1f4-8d84ed15971e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30fed99f-b5e6-4c12-a1f4-8d84ed15971e-workload-certs\") pod \"30fed99f-b5e6-4c12-a1f4-8d84ed15971e\" (UID: \"30fed99f-b5e6-4c12-a1f4-8d84ed15971e\") "
Oct 30 23:03:01.515617 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:01.515531    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30fed99f-b5e6-4c12-a1f4-8d84ed15971e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30fed99f-b5e6-4c12-a1f4-8d84ed15971e-credential-socket\") pod \"30fed99f-b5e6-4c12-a1f4-8d84ed15971e\" (UID: \"30fed99f-b5e6-4c12-a1f4-8d84ed15971e\") "
Oct 30 23:03:01.515617 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:01.515592    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30fed99f-b5e6-4c12-a1f4-8d84ed15971e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30fed99f-b5e6-4c12-a1f4-8d84ed15971e-workload-socket\") pod \"30fed99f-b5e6-4c12-a1f4-8d84ed15971e\" (UID: \"30fed99f-b5e6-4c12-a1f4-8d84ed15971e\") "
Oct 30 23:03:01.515728 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:01.515639    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30fed99f-b5e6-4c12-a1f4-8d84ed15971e-istiod-ca-cert" pod "30fed99f-b5e6-4c12-a1f4-8d84ed15971e" (UID: "30fed99f-b5e6-4c12-a1f4-8d84ed15971e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:03:01.515919 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:01.515897    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30fed99f-b5e6-4c12-a1f4-8d84ed15971e-workload-certs" pod "30fed99f-b5e6-4c12-a1f4-8d84ed15971e" (UID: "30fed99f-b5e6-4c12-a1f4-8d84ed15971e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:03:01.515996 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:01.515938    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30fed99f-b5e6-4c12-a1f4-8d84ed15971e-workload-socket" pod "30fed99f-b5e6-4c12-a1f4-8d84ed15971e" (UID: "30fed99f-b5e6-4c12-a1f4-8d84ed15971e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:03:01.516065 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:01.516007    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30fed99f-b5e6-4c12-a1f4-8d84ed15971e-istio-data" pod "30fed99f-b5e6-4c12-a1f4-8d84ed15971e" (UID: "30fed99f-b5e6-4c12-a1f4-8d84ed15971e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:03:01.516128 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:01.516111    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30fed99f-b5e6-4c12-a1f4-8d84ed15971e-credential-socket" pod "30fed99f-b5e6-4c12-a1f4-8d84ed15971e" (UID: "30fed99f-b5e6-4c12-a1f4-8d84ed15971e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:03:01.516297 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:01.516247    4477 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30fed99f-b5e6-4c12-a1f4-8d84ed15971e-istio-data\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:03:01.516501 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:01.516268    4477 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30fed99f-b5e6-4c12-a1f4-8d84ed15971e-workload-certs\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:03:01.516501 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:01.516453    4477 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30fed99f-b5e6-4c12-a1f4-8d84ed15971e-credential-socket\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:03:01.516501 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:01.516470    4477 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30fed99f-b5e6-4c12-a1f4-8d84ed15971e-workload-socket\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:03:01.516501 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:01.516483    4477 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30fed99f-b5e6-4c12-a1f4-8d84ed15971e-istiod-ca-cert\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:03:01.517707 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:01.517680    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30fed99f-b5e6-4c12-a1f4-8d84ed15971e-istio-envoy" pod "30fed99f-b5e6-4c12-a1f4-8d84ed15971e" (UID: "30fed99f-b5e6-4c12-a1f4-8d84ed15971e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:03:01.518021 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:01.517999    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30fed99f-b5e6-4c12-a1f4-8d84ed15971e-kube-api-access-j6dhb" pod "30fed99f-b5e6-4c12-a1f4-8d84ed15971e" (UID: "30fed99f-b5e6-4c12-a1f4-8d84ed15971e"). InnerVolumeSpecName "kube-api-access-j6dhb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:03:01.518323 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:01.518298    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/30fed99f-b5e6-4c12-a1f4-8d84ed15971e-istio-podinfo" pod "30fed99f-b5e6-4c12-a1f4-8d84ed15971e" (UID: "30fed99f-b5e6-4c12-a1f4-8d84ed15971e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:03:01.518416 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:01.518320    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30fed99f-b5e6-4c12-a1f4-8d84ed15971e-istio-token" pod "30fed99f-b5e6-4c12-a1f4-8d84ed15971e" (UID: "30fed99f-b5e6-4c12-a1f4-8d84ed15971e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:03:01.617474 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:01.617435    4477 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30fed99f-b5e6-4c12-a1f4-8d84ed15971e-istio-envoy\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:03:01.617474 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:01.617473    4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6dhb\" (UniqueName: \"kubernetes.io/projected/30fed99f-b5e6-4c12-a1f4-8d84ed15971e-kube-api-access-j6dhb\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:03:01.617474 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:01.617483    4477 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30fed99f-b5e6-4c12-a1f4-8d84ed15971e-istio-token\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:03:01.617682 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:01.617492    4477 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30fed99f-b5e6-4c12-a1f4-8d84ed15971e-istio-podinfo\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:03:01.806153 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:01.806083    4477 generic.go:356] "Generic (PLEG): container finished" podID="30fed99f-b5e6-4c12-a1f4-8d84ed15971e" containerID="0781d508d5529cb9d8676b66eafa6284ea19d8b76451ac40ce1aac9910076060" exitCode=0
Oct 30 23:03:01.806153 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:01.806132    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-bmccc"
Oct 30 23:03:01.806314 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:01.806168    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-bmccc" event={"ID":"30fed99f-b5e6-4c12-a1f4-8d84ed15971e","Type":"ContainerDied","Data":"0781d508d5529cb9d8676b66eafa6284ea19d8b76451ac40ce1aac9910076060"}
Oct 30 23:03:01.806314 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:01.806227    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-bmccc" event={"ID":"30fed99f-b5e6-4c12-a1f4-8d84ed15971e","Type":"ContainerDied","Data":"6b9c41ce39a073f91a5e82778075dc3f4e32af70ef63d09e6a955a3cbf207335"}
Oct 30 23:03:01.806314 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:01.806258    4477 scope.go:122] "RemoveContainer" containerID="0781d508d5529cb9d8676b66eafa6284ea19d8b76451ac40ce1aac9910076060"
Oct 30 23:03:01.816976 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:01.816960    4477 scope.go:122] "RemoveContainer" containerID="0781d508d5529cb9d8676b66eafa6284ea19d8b76451ac40ce1aac9910076060"
Oct 30 23:03:01.817263 ip-10-0-130-77 kubenswrapper[4477]: E1030 23:03:01.817225    4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0781d508d5529cb9d8676b66eafa6284ea19d8b76451ac40ce1aac9910076060\": container with ID starting with 0781d508d5529cb9d8676b66eafa6284ea19d8b76451ac40ce1aac9910076060 not found: ID does not exist" containerID="0781d508d5529cb9d8676b66eafa6284ea19d8b76451ac40ce1aac9910076060"
Oct 30 23:03:01.817327 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:01.817273    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0781d508d5529cb9d8676b66eafa6284ea19d8b76451ac40ce1aac9910076060"} err="failed to get container status \"0781d508d5529cb9d8676b66eafa6284ea19d8b76451ac40ce1aac9910076060\": rpc error: code = NotFound desc = could not find container \"0781d508d5529cb9d8676b66eafa6284ea19d8b76451ac40ce1aac9910076060\": container with ID starting with 0781d508d5529cb9d8676b66eafa6284ea19d8b76451ac40ce1aac9910076060 not found: ID does not exist"
Oct 30 23:03:01.859212 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:01.859189    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-bmccc"]
Oct 30 23:03:01.869370 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:01.869346    4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-3cd77f-openshift-default-6d76d5595c-bmccc"]
Oct 30 23:03:02.202899 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:02.202866    4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30fed99f-b5e6-4c12-a1f4-8d84ed15971e" path="/var/lib/kubelet/pods/30fed99f-b5e6-4c12-a1f4-8d84ed15971e/volumes"
Oct 30 23:03:02.308322 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:02.308289    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-cdfa86-9b956fcf5-z6t2f"]
Oct 30 23:03:02.308562 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:02.308535    4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cdfa86-9b956fcf5-z6t2f" podUID="401f33fa-f0da-4809-af95-3f4d30ebc830" containerName="maas-api" containerID="cri-o://ea3ef4b8d735d2d2c6ad8b31d5bfa268a0b07aea6257484f1567a28e43c09b21" gracePeriod=30
Oct 30 23:03:02.622120 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:02.622094    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cdfa86-9b956fcf5-z6t2f"
Oct 30 23:03:02.730455 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:02.730416    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/401f33fa-f0da-4809-af95-3f4d30ebc830-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/401f33fa-f0da-4809-af95-3f4d30ebc830-maas-api-tls\") pod \"401f33fa-f0da-4809-af95-3f4d30ebc830\" (UID: \"401f33fa-f0da-4809-af95-3f4d30ebc830\") "
Oct 30 23:03:02.730624 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:02.730602    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/401f33fa-f0da-4809-af95-3f4d30ebc830-kube-api-access-7kq8f\" (UniqueName: \"kubernetes.io/projected/401f33fa-f0da-4809-af95-3f4d30ebc830-kube-api-access-7kq8f\") pod \"401f33fa-f0da-4809-af95-3f4d30ebc830\" (UID: \"401f33fa-f0da-4809-af95-3f4d30ebc830\") "
Oct 30 23:03:02.730685 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:02.730654    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/401f33fa-f0da-4809-af95-3f4d30ebc830-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/401f33fa-f0da-4809-af95-3f4d30ebc830-openshift-service-ca\") pod \"401f33fa-f0da-4809-af95-3f4d30ebc830\" (UID: \"401f33fa-f0da-4809-af95-3f4d30ebc830\") "
Oct 30 23:03:02.730803 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:02.730779    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/401f33fa-f0da-4809-af95-3f4d30ebc830-metrics-certs\" (UniqueName: \"kubernetes.io/secret/401f33fa-f0da-4809-af95-3f4d30ebc830-metrics-certs\") pod \"401f33fa-f0da-4809-af95-3f4d30ebc830\" (UID: \"401f33fa-f0da-4809-af95-3f4d30ebc830\") "
Oct 30 23:03:02.731115 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:02.731088    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/401f33fa-f0da-4809-af95-3f4d30ebc830-openshift-service-ca" pod "401f33fa-f0da-4809-af95-3f4d30ebc830" (UID: "401f33fa-f0da-4809-af95-3f4d30ebc830"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:03:02.731387 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:02.731362    4477 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/401f33fa-f0da-4809-af95-3f4d30ebc830-openshift-service-ca\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:03:02.732762 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:02.732734    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/401f33fa-f0da-4809-af95-3f4d30ebc830-maas-api-tls" pod "401f33fa-f0da-4809-af95-3f4d30ebc830" (UID: "401f33fa-f0da-4809-af95-3f4d30ebc830"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:03:02.732849 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:02.732741    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/401f33fa-f0da-4809-af95-3f4d30ebc830-kube-api-access-7kq8f" pod "401f33fa-f0da-4809-af95-3f4d30ebc830" (UID: "401f33fa-f0da-4809-af95-3f4d30ebc830"). InnerVolumeSpecName "kube-api-access-7kq8f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:03:02.733080 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:02.733033    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/401f33fa-f0da-4809-af95-3f4d30ebc830-metrics-certs" pod "401f33fa-f0da-4809-af95-3f4d30ebc830" (UID: "401f33fa-f0da-4809-af95-3f4d30ebc830"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:03:02.827120 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:02.826998    4477 generic.go:356] "Generic (PLEG): container finished" podID="401f33fa-f0da-4809-af95-3f4d30ebc830" containerID="ea3ef4b8d735d2d2c6ad8b31d5bfa268a0b07aea6257484f1567a28e43c09b21" exitCode=0
Oct 30 23:03:02.827331 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:02.827311    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cdfa86-9b956fcf5-z6t2f"
Oct 30 23:03:02.828885 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:02.828108    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cdfa86-9b956fcf5-z6t2f" event={"ID":"401f33fa-f0da-4809-af95-3f4d30ebc830","Type":"ContainerDied","Data":"ea3ef4b8d735d2d2c6ad8b31d5bfa268a0b07aea6257484f1567a28e43c09b21"}
Oct 30 23:03:02.828885 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:02.828170    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-cdfa86-9b956fcf5-z6t2f" event={"ID":"401f33fa-f0da-4809-af95-3f4d30ebc830","Type":"ContainerDied","Data":"457f812cd120938b8cbb9f69c278cdd6029fc0564684a414722e5f4dd5c285f6"}
Oct 30 23:03:02.828885 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:02.828202    4477 scope.go:122] "RemoveContainer" containerID="ea3ef4b8d735d2d2c6ad8b31d5bfa268a0b07aea6257484f1567a28e43c09b21"
Oct 30 23:03:02.832764 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:02.832722    4477 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/401f33fa-f0da-4809-af95-3f4d30ebc830-metrics-certs\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:03:02.832764 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:02.832751    4477 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/401f33fa-f0da-4809-af95-3f4d30ebc830-maas-api-tls\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:03:02.832764 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:02.832764    4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kq8f\" (UniqueName: \"kubernetes.io/projected/401f33fa-f0da-4809-af95-3f4d30ebc830-kube-api-access-7kq8f\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:03:02.863889 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:02.863854    4477 scope.go:122] "RemoveContainer" containerID="ea3ef4b8d735d2d2c6ad8b31d5bfa268a0b07aea6257484f1567a28e43c09b21"
Oct 30 23:03:02.864290 ip-10-0-130-77 kubenswrapper[4477]: E1030 23:03:02.864263    4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea3ef4b8d735d2d2c6ad8b31d5bfa268a0b07aea6257484f1567a28e43c09b21\": container with ID starting with ea3ef4b8d735d2d2c6ad8b31d5bfa268a0b07aea6257484f1567a28e43c09b21 not found: ID does not exist" containerID="ea3ef4b8d735d2d2c6ad8b31d5bfa268a0b07aea6257484f1567a28e43c09b21"
Oct 30 23:03:02.864439 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:02.864303    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea3ef4b8d735d2d2c6ad8b31d5bfa268a0b07aea6257484f1567a28e43c09b21"} err="failed to get container status \"ea3ef4b8d735d2d2c6ad8b31d5bfa268a0b07aea6257484f1567a28e43c09b21\": rpc error: code = NotFound desc = could not find container \"ea3ef4b8d735d2d2c6ad8b31d5bfa268a0b07aea6257484f1567a28e43c09b21\": container with ID starting with ea3ef4b8d735d2d2c6ad8b31d5bfa268a0b07aea6257484f1567a28e43c09b21 not found: ID does not exist"
Oct 30 23:03:02.883133 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:02.883104    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-cdfa86-9b956fcf5-z6t2f"]
Oct 30 23:03:02.888260 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:02.888230    4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-cdfa86-9b956fcf5-z6t2f"]
Oct 30 23:03:04.194003 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:04.193968    4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="401f33fa-f0da-4809-af95-3f4d30ebc830" path="/var/lib/kubelet/pods/401f33fa-f0da-4809-af95-3f4d30ebc830/volumes"
Oct 30 23:03:19.863352 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:19.863312    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-76d54a-69f954c4cd-kxw27"]
Oct 30 23:03:19.864328 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:19.864290    4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-76d54a-69f954c4cd-kxw27" podUID="c17cee11-e2ac-469a-8759-c595e2e73191" containerName="maas-api" containerID="cri-o://1777cad3317a72488d0f89aee6aebcee82b8fc969df44539839cefe007cd622d" gracePeriod=30
Oct 30 23:03:20.208550 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:20.208525    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-76d54a-69f954c4cd-kxw27"
Oct 30 23:03:20.282374 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:20.282336    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c17cee11-e2ac-469a-8759-c595e2e73191-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c17cee11-e2ac-469a-8759-c595e2e73191-metrics-certs\") pod \"c17cee11-e2ac-469a-8759-c595e2e73191\" (UID: \"c17cee11-e2ac-469a-8759-c595e2e73191\") "
Oct 30 23:03:20.282556 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:20.282415    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c17cee11-e2ac-469a-8759-c595e2e73191-kube-api-access-ckzcj\" (UniqueName: \"kubernetes.io/projected/c17cee11-e2ac-469a-8759-c595e2e73191-kube-api-access-ckzcj\") pod \"c17cee11-e2ac-469a-8759-c595e2e73191\" (UID: \"c17cee11-e2ac-469a-8759-c595e2e73191\") "
Oct 30 23:03:20.282634 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:20.282595    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c17cee11-e2ac-469a-8759-c595e2e73191-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c17cee11-e2ac-469a-8759-c595e2e73191-openshift-service-ca\") pod \"c17cee11-e2ac-469a-8759-c595e2e73191\" (UID: \"c17cee11-e2ac-469a-8759-c595e2e73191\") "
Oct 30 23:03:20.284540 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:20.282732    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c17cee11-e2ac-469a-8759-c595e2e73191-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c17cee11-e2ac-469a-8759-c595e2e73191-maas-api-tls\") pod \"c17cee11-e2ac-469a-8759-c595e2e73191\" (UID: \"c17cee11-e2ac-469a-8759-c595e2e73191\") "
Oct 30 23:03:20.284698 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:20.284616    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c17cee11-e2ac-469a-8759-c595e2e73191-openshift-service-ca" pod "c17cee11-e2ac-469a-8759-c595e2e73191" (UID: "c17cee11-e2ac-469a-8759-c595e2e73191"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:03:20.287259 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:20.287215    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c17cee11-e2ac-469a-8759-c595e2e73191-maas-api-tls" pod "c17cee11-e2ac-469a-8759-c595e2e73191" (UID: "c17cee11-e2ac-469a-8759-c595e2e73191"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:03:20.287390 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:20.287334    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c17cee11-e2ac-469a-8759-c595e2e73191-metrics-certs" pod "c17cee11-e2ac-469a-8759-c595e2e73191" (UID: "c17cee11-e2ac-469a-8759-c595e2e73191"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:03:20.287561 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:20.287539    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c17cee11-e2ac-469a-8759-c595e2e73191-kube-api-access-ckzcj" pod "c17cee11-e2ac-469a-8759-c595e2e73191" (UID: "c17cee11-e2ac-469a-8759-c595e2e73191"). InnerVolumeSpecName "kube-api-access-ckzcj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:03:20.394086 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:20.394029    4477 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c17cee11-e2ac-469a-8759-c595e2e73191-openshift-service-ca\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:03:20.394086 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:20.394089    4477 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c17cee11-e2ac-469a-8759-c595e2e73191-maas-api-tls\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:03:20.394398 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:20.394105    4477 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c17cee11-e2ac-469a-8759-c595e2e73191-metrics-certs\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:03:20.394398 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:20.394118    4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckzcj\" (UniqueName: \"kubernetes.io/projected/c17cee11-e2ac-469a-8759-c595e2e73191-kube-api-access-ckzcj\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:03:20.661980 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:20.661942    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-012ba0/distinct-2-w6-012ba0-kserve-78dbcd8c46-hqdfm"]
Oct 30 23:03:20.679759 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:20.679731    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-012ba0/premium-w6-012ba0-kserve-658654dbdb-d2k27"]
Oct 30 23:03:20.946956 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:20.946868    4477 generic.go:356] "Generic (PLEG): container finished" podID="c17cee11-e2ac-469a-8759-c595e2e73191" containerID="1777cad3317a72488d0f89aee6aebcee82b8fc969df44539839cefe007cd622d" exitCode=0
Oct 30 23:03:20.946956 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:20.946918    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-76d54a-69f954c4cd-kxw27"
Oct 30 23:03:20.947448 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:20.946954    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-76d54a-69f954c4cd-kxw27" event={"ID":"c17cee11-e2ac-469a-8759-c595e2e73191","Type":"ContainerDied","Data":"1777cad3317a72488d0f89aee6aebcee82b8fc969df44539839cefe007cd622d"}
Oct 30 23:03:20.947448 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:20.947011    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-76d54a-69f954c4cd-kxw27" event={"ID":"c17cee11-e2ac-469a-8759-c595e2e73191","Type":"ContainerDied","Data":"22df9a1907c4403e844bc151fcb28a8adf1f1d51648831f046ef407efffeaa42"}
Oct 30 23:03:20.947448 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:20.947033    4477 scope.go:122] "RemoveContainer" containerID="1777cad3317a72488d0f89aee6aebcee82b8fc969df44539839cefe007cd622d"
Oct 30 23:03:20.957874 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:20.957855    4477 scope.go:122] "RemoveContainer" containerID="1777cad3317a72488d0f89aee6aebcee82b8fc969df44539839cefe007cd622d"
Oct 30 23:03:20.958164 ip-10-0-130-77 kubenswrapper[4477]: E1030 23:03:20.958141    4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1777cad3317a72488d0f89aee6aebcee82b8fc969df44539839cefe007cd622d\": container with ID starting with 1777cad3317a72488d0f89aee6aebcee82b8fc969df44539839cefe007cd622d not found: ID does not exist" containerID="1777cad3317a72488d0f89aee6aebcee82b8fc969df44539839cefe007cd622d"
Oct 30 23:03:20.958281 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:20.958168    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1777cad3317a72488d0f89aee6aebcee82b8fc969df44539839cefe007cd622d"} err="failed to get container status \"1777cad3317a72488d0f89aee6aebcee82b8fc969df44539839cefe007cd622d\": rpc error: code = NotFound desc = could not find container \"1777cad3317a72488d0f89aee6aebcee82b8fc969df44539839cefe007cd622d\": container with ID starting with 1777cad3317a72488d0f89aee6aebcee82b8fc969df44539839cefe007cd622d not found: ID does not exist"
Oct 30 23:03:20.994124 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:20.994102    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-76d54a-69f954c4cd-kxw27"]
Oct 30 23:03:21.000338 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:21.000313    4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-76d54a-69f954c4cd-kxw27"]
Oct 30 23:03:22.126064 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:22.125868    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-510f80-5475f9dfb-njms4"]
Oct 30 23:03:22.126641 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:22.126208    4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-510f80-5475f9dfb-njms4" podUID="00783ce5-3f4a-4649-9c93-28b8a1c01b1e" containerName="maas-api" containerID="cri-o://fe5de093d371f6396f53000157af70d6ccde6be912b369ffbd4bef41896ab151" gracePeriod=30
Oct 30 23:03:22.203081 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:22.202185    4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c17cee11-e2ac-469a-8759-c595e2e73191" path="/var/lib/kubelet/pods/c17cee11-e2ac-469a-8759-c595e2e73191/volumes"
Oct 30 23:03:22.460865 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:22.460835    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-510f80-5475f9dfb-njms4"
Oct 30 23:03:22.520863 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:22.520459    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00783ce5-3f4a-4649-9c93-28b8a1c01b1e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00783ce5-3f4a-4649-9c93-28b8a1c01b1e-maas-api-tls\") pod \"00783ce5-3f4a-4649-9c93-28b8a1c01b1e\" (UID: \"00783ce5-3f4a-4649-9c93-28b8a1c01b1e\") "
Oct 30 23:03:22.520863 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:22.520543    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00783ce5-3f4a-4649-9c93-28b8a1c01b1e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/00783ce5-3f4a-4649-9c93-28b8a1c01b1e-metrics-certs\") pod \"00783ce5-3f4a-4649-9c93-28b8a1c01b1e\" (UID: \"00783ce5-3f4a-4649-9c93-28b8a1c01b1e\") "
Oct 30 23:03:22.520863 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:22.520652    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00783ce5-3f4a-4649-9c93-28b8a1c01b1e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00783ce5-3f4a-4649-9c93-28b8a1c01b1e-openshift-service-ca\") pod \"00783ce5-3f4a-4649-9c93-28b8a1c01b1e\" (UID: \"00783ce5-3f4a-4649-9c93-28b8a1c01b1e\") "
Oct 30 23:03:22.520863 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:22.520868    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00783ce5-3f4a-4649-9c93-28b8a1c01b1e-kube-api-access-sv86w\" (UniqueName: \"kubernetes.io/projected/00783ce5-3f4a-4649-9c93-28b8a1c01b1e-kube-api-access-sv86w\") pod \"00783ce5-3f4a-4649-9c93-28b8a1c01b1e\" (UID: \"00783ce5-3f4a-4649-9c93-28b8a1c01b1e\") "
Oct 30 23:03:22.522135 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:22.522098    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00783ce5-3f4a-4649-9c93-28b8a1c01b1e-openshift-service-ca" pod "00783ce5-3f4a-4649-9c93-28b8a1c01b1e" (UID: "00783ce5-3f4a-4649-9c93-28b8a1c01b1e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:03:22.524583 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:22.524550    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00783ce5-3f4a-4649-9c93-28b8a1c01b1e-maas-api-tls" pod "00783ce5-3f4a-4649-9c93-28b8a1c01b1e" (UID: "00783ce5-3f4a-4649-9c93-28b8a1c01b1e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:03:22.525174 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:22.525147    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00783ce5-3f4a-4649-9c93-28b8a1c01b1e-metrics-certs" pod "00783ce5-3f4a-4649-9c93-28b8a1c01b1e" (UID: "00783ce5-3f4a-4649-9c93-28b8a1c01b1e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:03:22.526697 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:22.526668    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00783ce5-3f4a-4649-9c93-28b8a1c01b1e-kube-api-access-sv86w" pod "00783ce5-3f4a-4649-9c93-28b8a1c01b1e" (UID: "00783ce5-3f4a-4649-9c93-28b8a1c01b1e"). InnerVolumeSpecName "kube-api-access-sv86w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:03:22.622917 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:22.622877    4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sv86w\" (UniqueName: \"kubernetes.io/projected/00783ce5-3f4a-4649-9c93-28b8a1c01b1e-kube-api-access-sv86w\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:03:22.622917 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:22.622916    4477 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00783ce5-3f4a-4649-9c93-28b8a1c01b1e-maas-api-tls\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:03:22.622917 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:22.622932    4477 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00783ce5-3f4a-4649-9c93-28b8a1c01b1e-metrics-certs\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:03:22.623259 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:22.622944    4477 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00783ce5-3f4a-4649-9c93-28b8a1c01b1e-openshift-service-ca\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:03:22.992100 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:22.990414    4477 generic.go:356] "Generic (PLEG): container finished" podID="00783ce5-3f4a-4649-9c93-28b8a1c01b1e" containerID="fe5de093d371f6396f53000157af70d6ccde6be912b369ffbd4bef41896ab151" exitCode=0
Oct 30 23:03:22.992100 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:22.990592    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-510f80-5475f9dfb-njms4"
Oct 30 23:03:22.992100 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:22.990669    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-510f80-5475f9dfb-njms4" event={"ID":"00783ce5-3f4a-4649-9c93-28b8a1c01b1e","Type":"ContainerDied","Data":"fe5de093d371f6396f53000157af70d6ccde6be912b369ffbd4bef41896ab151"}
Oct 30 23:03:22.992100 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:22.990724    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-510f80-5475f9dfb-njms4" event={"ID":"00783ce5-3f4a-4649-9c93-28b8a1c01b1e","Type":"ContainerDied","Data":"5e8e3a0b01e62f4b0fcc77be3f7d7c845f40b18f6c8ac52a041073c638e9a0fe"}
Oct 30 23:03:22.992100 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:22.990752    4477 scope.go:122] "RemoveContainer" containerID="fe5de093d371f6396f53000157af70d6ccde6be912b369ffbd4bef41896ab151"
Oct 30 23:03:23.005156 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:23.005137    4477 scope.go:122] "RemoveContainer" containerID="fe5de093d371f6396f53000157af70d6ccde6be912b369ffbd4bef41896ab151"
Oct 30 23:03:23.005438 ip-10-0-130-77 kubenswrapper[4477]: E1030 23:03:23.005419    4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe5de093d371f6396f53000157af70d6ccde6be912b369ffbd4bef41896ab151\": container with ID starting with fe5de093d371f6396f53000157af70d6ccde6be912b369ffbd4bef41896ab151 not found: ID does not exist" containerID="fe5de093d371f6396f53000157af70d6ccde6be912b369ffbd4bef41896ab151"
Oct 30 23:03:23.005515 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:23.005447    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe5de093d371f6396f53000157af70d6ccde6be912b369ffbd4bef41896ab151"} err="failed to get container status \"fe5de093d371f6396f53000157af70d6ccde6be912b369ffbd4bef41896ab151\": rpc error: code = NotFound desc = could not find container \"fe5de093d371f6396f53000157af70d6ccde6be912b369ffbd4bef41896ab151\": container with ID starting with fe5de093d371f6396f53000157af70d6ccde6be912b369ffbd4bef41896ab151 not found: ID does not exist"
Oct 30 23:03:23.043394 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:23.043368    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-510f80-5475f9dfb-njms4"]
Oct 30 23:03:23.047124 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:23.047098    4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-510f80-5475f9dfb-njms4"]
Oct 30 23:03:24.193099 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:24.193028    4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00783ce5-3f4a-4649-9c93-28b8a1c01b1e" path="/var/lib/kubelet/pods/00783ce5-3f4a-4649-9c93-28b8a1c01b1e/volumes"
Oct 30 23:03:26.267961 ip-10-0-130-77 kubenswrapper[4477]: E1030 23:03:26.267927    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 23:03:35.684362 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:35.684321    4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-012ba0/distinct-2-w6-012ba0-kserve-78dbcd8c46-hqdfm" podUID="8326b014-989b-4722-befb-6fa8ea60688b" containerName="main" containerID="cri-o://8755ce6e726a15081433d3a94bac9ad168a0241c8a88446a91721ed2bd38588c" gracePeriod=45
Oct 30 23:03:35.700412 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:35.700376    4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-012ba0/premium-w6-012ba0-kserve-658654dbdb-d2k27" podUID="085847e5-4f50-419e-984f-b612b104ed90" containerName="main" containerID="cri-o://e546755e29e463e07191b8d4ffe3ca38d37c8980123729aa9b5621eb4939de7f" gracePeriod=45
Oct 30 23:03:35.853071 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:35.852765    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6bz92"]
Oct 30 23:03:36.012184 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:36.012153    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-012ba0/distinct-2-w6-012ba0-kserve-78dbcd8c46-hqdfm"
Oct 30 23:03:36.018828 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:36.018803    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-012ba0/premium-w6-012ba0-kserve-658654dbdb-d2k27"
Oct 30 23:03:36.031671 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:36.031639    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-54114e/premium-w5-54114e-kserve-57f6747bcc-7wjrw"]
Oct 30 23:03:36.082879 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:36.082841    4477 generic.go:356] "Generic (PLEG): container finished" podID="085847e5-4f50-419e-984f-b612b104ed90" containerID="e546755e29e463e07191b8d4ffe3ca38d37c8980123729aa9b5621eb4939de7f" exitCode=0
Oct 30 23:03:36.083111 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:36.082891    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-012ba0/premium-w6-012ba0-kserve-658654dbdb-d2k27"
Oct 30 23:03:36.083111 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:36.082933    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-012ba0/premium-w6-012ba0-kserve-658654dbdb-d2k27" event={"ID":"085847e5-4f50-419e-984f-b612b104ed90","Type":"ContainerDied","Data":"e546755e29e463e07191b8d4ffe3ca38d37c8980123729aa9b5621eb4939de7f"}
Oct 30 23:03:36.083111 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:36.082973    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-012ba0/premium-w6-012ba0-kserve-658654dbdb-d2k27" event={"ID":"085847e5-4f50-419e-984f-b612b104ed90","Type":"ContainerDied","Data":"ed317ea5f2fa3157b21723015a25614a78865bea0b31787b991fefe5ca36cdd2"}
Oct 30 23:03:36.083111 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:36.083011    4477 scope.go:122] "RemoveContainer" containerID="e546755e29e463e07191b8d4ffe3ca38d37c8980123729aa9b5621eb4939de7f"
Oct 30 23:03:36.085193 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:36.085168    4477 generic.go:356] "Generic (PLEG): container finished" podID="8326b014-989b-4722-befb-6fa8ea60688b" containerID="8755ce6e726a15081433d3a94bac9ad168a0241c8a88446a91721ed2bd38588c" exitCode=0
Oct 30 23:03:36.085308 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:36.085201    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-012ba0/distinct-2-w6-012ba0-kserve-78dbcd8c46-hqdfm"
Oct 30 23:03:36.085308 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:36.085251    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-012ba0/distinct-2-w6-012ba0-kserve-78dbcd8c46-hqdfm" event={"ID":"8326b014-989b-4722-befb-6fa8ea60688b","Type":"ContainerDied","Data":"8755ce6e726a15081433d3a94bac9ad168a0241c8a88446a91721ed2bd38588c"}
Oct 30 23:03:36.085308 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:36.085302    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-012ba0/distinct-2-w6-012ba0-kserve-78dbcd8c46-hqdfm" event={"ID":"8326b014-989b-4722-befb-6fa8ea60688b","Type":"ContainerDied","Data":"19b77357627087ee3e9bea6fe79f2df5b62b276843970bff85ef10cb793be06d"}
Oct 30 23:03:36.097085 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:36.092591    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/085847e5-4f50-419e-984f-b612b104ed90-dshm\" (UniqueName: \"kubernetes.io/empty-dir/085847e5-4f50-419e-984f-b612b104ed90-dshm\") pod \"085847e5-4f50-419e-984f-b612b104ed90\" (UID: \"085847e5-4f50-419e-984f-b612b104ed90\") "
Oct 30 23:03:36.097085 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:36.092654    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8326b014-989b-4722-befb-6fa8ea60688b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8326b014-989b-4722-befb-6fa8ea60688b-model-cache\") pod \"8326b014-989b-4722-befb-6fa8ea60688b\" (UID: \"8326b014-989b-4722-befb-6fa8ea60688b\") "
Oct 30 23:03:36.097085 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:36.092718    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8326b014-989b-4722-befb-6fa8ea60688b-tls-certs\" (UniqueName: \"kubernetes.io/secret/8326b014-989b-4722-befb-6fa8ea60688b-tls-certs\") pod \"8326b014-989b-4722-befb-6fa8ea60688b\" (UID: \"8326b014-989b-4722-befb-6fa8ea60688b\") "
Oct 30 23:03:36.097085 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:36.092766    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8326b014-989b-4722-befb-6fa8ea60688b-home\" (UniqueName: \"kubernetes.io/empty-dir/8326b014-989b-4722-befb-6fa8ea60688b-home\") pod \"8326b014-989b-4722-befb-6fa8ea60688b\" (UID: \"8326b014-989b-4722-befb-6fa8ea60688b\") "
Oct 30 23:03:36.097085 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:36.092800    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8326b014-989b-4722-befb-6fa8ea60688b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8326b014-989b-4722-befb-6fa8ea60688b-tmp-dir\") pod \"8326b014-989b-4722-befb-6fa8ea60688b\" (UID: \"8326b014-989b-4722-befb-6fa8ea60688b\") "
Oct 30 23:03:36.097085 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:36.092852    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/085847e5-4f50-419e-984f-b612b104ed90-kube-api-access-nnclv\" (UniqueName: \"kubernetes.io/projected/085847e5-4f50-419e-984f-b612b104ed90-kube-api-access-nnclv\") pod \"085847e5-4f50-419e-984f-b612b104ed90\" (UID: \"085847e5-4f50-419e-984f-b612b104ed90\") "
Oct 30 23:03:36.097085 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:36.092910    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/085847e5-4f50-419e-984f-b612b104ed90-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/085847e5-4f50-419e-984f-b612b104ed90-model-cache\") pod \"085847e5-4f50-419e-984f-b612b104ed90\" (UID: \"085847e5-4f50-419e-984f-b612b104ed90\") "
Oct 30 23:03:36.097085 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:36.092959    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8326b014-989b-4722-befb-6fa8ea60688b-kube-api-access-g7xr2\" (UniqueName: \"kubernetes.io/projected/8326b014-989b-4722-befb-6fa8ea60688b-kube-api-access-g7xr2\") pod \"8326b014-989b-4722-befb-6fa8ea60688b\" (UID: \"8326b014-989b-4722-befb-6fa8ea60688b\") "
Oct 30 23:03:36.097085 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:36.092993    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/085847e5-4f50-419e-984f-b612b104ed90-home\" (UniqueName: \"kubernetes.io/empty-dir/085847e5-4f50-419e-984f-b612b104ed90-home\") pod \"085847e5-4f50-419e-984f-b612b104ed90\" (UID: \"085847e5-4f50-419e-984f-b612b104ed90\") "
Oct 30 23:03:36.097085 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:36.093033    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/085847e5-4f50-419e-984f-b612b104ed90-tls-certs\" (UniqueName: \"kubernetes.io/secret/085847e5-4f50-419e-984f-b612b104ed90-tls-certs\") pod \"085847e5-4f50-419e-984f-b612b104ed90\" (UID: \"085847e5-4f50-419e-984f-b612b104ed90\") "
Oct 30 23:03:36.097085 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:36.093108    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8326b014-989b-4722-befb-6fa8ea60688b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8326b014-989b-4722-befb-6fa8ea60688b-dshm\") pod \"8326b014-989b-4722-befb-6fa8ea60688b\" (UID: \"8326b014-989b-4722-befb-6fa8ea60688b\") "
Oct 30 23:03:36.097085 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:36.093179    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/085847e5-4f50-419e-984f-b612b104ed90-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/085847e5-4f50-419e-984f-b612b104ed90-tmp-dir\") pod \"085847e5-4f50-419e-984f-b612b104ed90\" (UID: \"085847e5-4f50-419e-984f-b612b104ed90\") "
Oct 30 23:03:36.097085 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:36.096111    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/085847e5-4f50-419e-984f-b612b104ed90-tmp-dir" pod "085847e5-4f50-419e-984f-b612b104ed90" (UID: "085847e5-4f50-419e-984f-b612b104ed90"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:03:36.097085 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:36.096311    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/085847e5-4f50-419e-984f-b612b104ed90-model-cache" pod "085847e5-4f50-419e-984f-b612b104ed90" (UID: "085847e5-4f50-419e-984f-b612b104ed90"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:03:36.112291 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:36.112152    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8326b014-989b-4722-befb-6fa8ea60688b-home" pod "8326b014-989b-4722-befb-6fa8ea60688b" (UID: "8326b014-989b-4722-befb-6fa8ea60688b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:03:36.113681 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:36.113619    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/085847e5-4f50-419e-984f-b612b104ed90-home" pod "085847e5-4f50-419e-984f-b612b104ed90" (UID: "085847e5-4f50-419e-984f-b612b104ed90"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:03:36.113681 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:36.113671    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8326b014-989b-4722-befb-6fa8ea60688b-tmp-dir" pod "8326b014-989b-4722-befb-6fa8ea60688b" (UID: "8326b014-989b-4722-befb-6fa8ea60688b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:03:36.114023 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:36.113986    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8326b014-989b-4722-befb-6fa8ea60688b-model-cache" pod "8326b014-989b-4722-befb-6fa8ea60688b" (UID: "8326b014-989b-4722-befb-6fa8ea60688b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:03:36.118378 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:36.118346    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8326b014-989b-4722-befb-6fa8ea60688b-dshm" pod "8326b014-989b-4722-befb-6fa8ea60688b" (UID: "8326b014-989b-4722-befb-6fa8ea60688b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:03:36.120681 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:36.120655    4477 scope.go:122] "RemoveContainer" containerID="e546755e29e463e07191b8d4ffe3ca38d37c8980123729aa9b5621eb4939de7f"
Oct 30 23:03:36.121099 ip-10-0-130-77 kubenswrapper[4477]: E1030 23:03:36.121034    4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e546755e29e463e07191b8d4ffe3ca38d37c8980123729aa9b5621eb4939de7f\": container with ID starting with e546755e29e463e07191b8d4ffe3ca38d37c8980123729aa9b5621eb4939de7f not found: ID does not exist" containerID="e546755e29e463e07191b8d4ffe3ca38d37c8980123729aa9b5621eb4939de7f"
Oct 30 23:03:36.121197 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:36.121113    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e546755e29e463e07191b8d4ffe3ca38d37c8980123729aa9b5621eb4939de7f"} err="failed to get container status \"e546755e29e463e07191b8d4ffe3ca38d37c8980123729aa9b5621eb4939de7f\": rpc error: code = NotFound desc = could not find container \"e546755e29e463e07191b8d4ffe3ca38d37c8980123729aa9b5621eb4939de7f\": container with ID starting with e546755e29e463e07191b8d4ffe3ca38d37c8980123729aa9b5621eb4939de7f not found: ID does not exist"
Oct 30 23:03:36.121197 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:36.121138    4477 scope.go:122] "RemoveContainer" containerID="8755ce6e726a15081433d3a94bac9ad168a0241c8a88446a91721ed2bd38588c"
Oct 30 23:03:36.123299 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:36.123275    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8326b014-989b-4722-befb-6fa8ea60688b-tls-certs" pod "8326b014-989b-4722-befb-6fa8ea60688b" (UID: "8326b014-989b-4722-befb-6fa8ea60688b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:03:36.123384 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:36.123304    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/085847e5-4f50-419e-984f-b612b104ed90-tls-certs" pod "085847e5-4f50-419e-984f-b612b104ed90" (UID: "085847e5-4f50-419e-984f-b612b104ed90"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:03:36.123384 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:36.123308    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/085847e5-4f50-419e-984f-b612b104ed90-kube-api-access-nnclv" pod "085847e5-4f50-419e-984f-b612b104ed90" (UID: "085847e5-4f50-419e-984f-b612b104ed90"). InnerVolumeSpecName "kube-api-access-nnclv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:03:36.123509 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:36.123410    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8326b014-989b-4722-befb-6fa8ea60688b-kube-api-access-g7xr2" pod "8326b014-989b-4722-befb-6fa8ea60688b" (UID: "8326b014-989b-4722-befb-6fa8ea60688b"). InnerVolumeSpecName "kube-api-access-g7xr2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:03:36.124641 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:36.124608    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/085847e5-4f50-419e-984f-b612b104ed90-dshm" pod "085847e5-4f50-419e-984f-b612b104ed90" (UID: "085847e5-4f50-419e-984f-b612b104ed90"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:03:36.130464 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:36.130438    4477 scope.go:122] "RemoveContainer" containerID="8755ce6e726a15081433d3a94bac9ad168a0241c8a88446a91721ed2bd38588c"
Oct 30 23:03:36.130742 ip-10-0-130-77 kubenswrapper[4477]: E1030 23:03:36.130721    4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8755ce6e726a15081433d3a94bac9ad168a0241c8a88446a91721ed2bd38588c\": container with ID starting with 8755ce6e726a15081433d3a94bac9ad168a0241c8a88446a91721ed2bd38588c not found: ID does not exist" containerID="8755ce6e726a15081433d3a94bac9ad168a0241c8a88446a91721ed2bd38588c"
Oct 30 23:03:36.130796 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:36.130753    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8755ce6e726a15081433d3a94bac9ad168a0241c8a88446a91721ed2bd38588c"} err="failed to get container status \"8755ce6e726a15081433d3a94bac9ad168a0241c8a88446a91721ed2bd38588c\": rpc error: code = NotFound desc = could not find container \"8755ce6e726a15081433d3a94bac9ad168a0241c8a88446a91721ed2bd38588c\": container with ID starting with 8755ce6e726a15081433d3a94bac9ad168a0241c8a88446a91721ed2bd38588c not found: ID does not exist"
Oct 30 23:03:36.195164 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:36.195132    4477 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/085847e5-4f50-419e-984f-b612b104ed90-model-cache\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:03:36.195164 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:36.195160    4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7xr2\" (UniqueName: \"kubernetes.io/projected/8326b014-989b-4722-befb-6fa8ea60688b-kube-api-access-g7xr2\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:03:36.195384 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:36.195176    4477 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/085847e5-4f50-419e-984f-b612b104ed90-home\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:03:36.195384 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:36.195189    4477 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/085847e5-4f50-419e-984f-b612b104ed90-tls-certs\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:03:36.195384 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:36.195200    4477 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8326b014-989b-4722-befb-6fa8ea60688b-dshm\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:03:36.195384 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:36.195211    4477 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/085847e5-4f50-419e-984f-b612b104ed90-tmp-dir\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:03:36.195384 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:36.195221    4477 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/085847e5-4f50-419e-984f-b612b104ed90-dshm\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:03:36.195384 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:36.195231    4477 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8326b014-989b-4722-befb-6fa8ea60688b-model-cache\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:03:36.195384 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:36.195244    4477 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8326b014-989b-4722-befb-6fa8ea60688b-tls-certs\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:03:36.195384 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:36.195254    4477 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8326b014-989b-4722-befb-6fa8ea60688b-home\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:03:36.195384 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:36.195264    4477 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8326b014-989b-4722-befb-6fa8ea60688b-tmp-dir\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:03:36.195384 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:36.195275    4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnclv\" (UniqueName: \"kubernetes.io/projected/085847e5-4f50-419e-984f-b612b104ed90-kube-api-access-nnclv\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:03:36.424246 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:36.424208    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-012ba0/premium-w6-012ba0-kserve-658654dbdb-d2k27"]
Oct 30 23:03:36.433465 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:36.433439    4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-012ba0/premium-w6-012ba0-kserve-658654dbdb-d2k27"]
Oct 30 23:03:36.455164 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:36.455134    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-012ba0/distinct-2-w6-012ba0-kserve-78dbcd8c46-hqdfm"]
Oct 30 23:03:36.457988 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:36.457960    4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-012ba0/distinct-2-w6-012ba0-kserve-78dbcd8c46-hqdfm"]
Oct 30 23:03:38.193294 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:38.193265    4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="085847e5-4f50-419e-984f-b612b104ed90" path="/var/lib/kubelet/pods/085847e5-4f50-419e-984f-b612b104ed90/volumes"
Oct 30 23:03:38.193758 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:38.193742    4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8326b014-989b-4722-befb-6fa8ea60688b" path="/var/lib/kubelet/pods/8326b014-989b-4722-befb-6fa8ea60688b/volumes"
Oct 30 23:03:44.387263 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:44.387232    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5983ea-586646597c-z222v"]
Oct 30 23:03:44.387660 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:44.387426    4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5983ea-586646597c-z222v" podUID="738731d9-e739-4c3e-8936-9c4d01f5fc04" containerName="maas-api" containerID="cri-o://af0ac40b526d5b6b4fa4a597572ff9e9d205c384db85782f0828ff46bd2c43b7" gracePeriod=30
Oct 30 23:03:44.734163 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:44.734135    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5983ea-586646597c-z222v"
Oct 30 23:03:44.796874 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:44.796844    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/738731d9-e739-4c3e-8936-9c4d01f5fc04-kube-api-access-58lwp\" (UniqueName: \"kubernetes.io/projected/738731d9-e739-4c3e-8936-9c4d01f5fc04-kube-api-access-58lwp\") pod \"738731d9-e739-4c3e-8936-9c4d01f5fc04\" (UID: \"738731d9-e739-4c3e-8936-9c4d01f5fc04\") "
Oct 30 23:03:44.797074 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:44.796890    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/738731d9-e739-4c3e-8936-9c4d01f5fc04-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/738731d9-e739-4c3e-8936-9c4d01f5fc04-maas-api-tls\") pod \"738731d9-e739-4c3e-8936-9c4d01f5fc04\" (UID: \"738731d9-e739-4c3e-8936-9c4d01f5fc04\") "
Oct 30 23:03:44.797161 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:44.797101    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/738731d9-e739-4c3e-8936-9c4d01f5fc04-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/738731d9-e739-4c3e-8936-9c4d01f5fc04-openshift-service-ca\") pod \"738731d9-e739-4c3e-8936-9c4d01f5fc04\" (UID: \"738731d9-e739-4c3e-8936-9c4d01f5fc04\") "
Oct 30 23:03:44.797161 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:44.797140    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/738731d9-e739-4c3e-8936-9c4d01f5fc04-metrics-certs\" (UniqueName: \"kubernetes.io/secret/738731d9-e739-4c3e-8936-9c4d01f5fc04-metrics-certs\") pod \"738731d9-e739-4c3e-8936-9c4d01f5fc04\" (UID: \"738731d9-e739-4c3e-8936-9c4d01f5fc04\") "
Oct 30 23:03:44.797546 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:44.797516    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/738731d9-e739-4c3e-8936-9c4d01f5fc04-openshift-service-ca" pod "738731d9-e739-4c3e-8936-9c4d01f5fc04" (UID: "738731d9-e739-4c3e-8936-9c4d01f5fc04"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:03:44.797666 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:44.797644    4477 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/738731d9-e739-4c3e-8936-9c4d01f5fc04-openshift-service-ca\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:03:44.799270 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:44.799240    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/738731d9-e739-4c3e-8936-9c4d01f5fc04-kube-api-access-58lwp" pod "738731d9-e739-4c3e-8936-9c4d01f5fc04" (UID: "738731d9-e739-4c3e-8936-9c4d01f5fc04"). InnerVolumeSpecName "kube-api-access-58lwp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:03:44.799579 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:44.799556    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/738731d9-e739-4c3e-8936-9c4d01f5fc04-maas-api-tls" pod "738731d9-e739-4c3e-8936-9c4d01f5fc04" (UID: "738731d9-e739-4c3e-8936-9c4d01f5fc04"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:03:44.799718 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:44.799703    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/738731d9-e739-4c3e-8936-9c4d01f5fc04-metrics-certs" pod "738731d9-e739-4c3e-8936-9c4d01f5fc04" (UID: "738731d9-e739-4c3e-8936-9c4d01f5fc04"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:03:44.903094 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:44.898995    4477 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/738731d9-e739-4c3e-8936-9c4d01f5fc04-metrics-certs\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:03:44.903094 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:44.899055    4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58lwp\" (UniqueName: \"kubernetes.io/projected/738731d9-e739-4c3e-8936-9c4d01f5fc04-kube-api-access-58lwp\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:03:44.903094 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:44.899074    4477 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/738731d9-e739-4c3e-8936-9c4d01f5fc04-maas-api-tls\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:03:45.144101 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:45.144066    4477 generic.go:356] "Generic (PLEG): container finished" podID="738731d9-e739-4c3e-8936-9c4d01f5fc04" containerID="af0ac40b526d5b6b4fa4a597572ff9e9d205c384db85782f0828ff46bd2c43b7" exitCode=0
Oct 30 23:03:45.144255 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:45.144110    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5983ea-586646597c-z222v"
Oct 30 23:03:45.144255 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:45.144135    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5983ea-586646597c-z222v" event={"ID":"738731d9-e739-4c3e-8936-9c4d01f5fc04","Type":"ContainerDied","Data":"af0ac40b526d5b6b4fa4a597572ff9e9d205c384db85782f0828ff46bd2c43b7"}
Oct 30 23:03:45.144255 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:45.144195    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5983ea-586646597c-z222v" event={"ID":"738731d9-e739-4c3e-8936-9c4d01f5fc04","Type":"ContainerDied","Data":"04d6d433fd6f668a0dc12e4808e7e1ffcf58d2b5b82662b5865bda2d897fd925"}
Oct 30 23:03:45.144255 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:45.144221    4477 scope.go:122] "RemoveContainer" containerID="af0ac40b526d5b6b4fa4a597572ff9e9d205c384db85782f0828ff46bd2c43b7"
Oct 30 23:03:45.154807 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:45.154789    4477 scope.go:122] "RemoveContainer" containerID="af0ac40b526d5b6b4fa4a597572ff9e9d205c384db85782f0828ff46bd2c43b7"
Oct 30 23:03:45.155096 ip-10-0-130-77 kubenswrapper[4477]: E1030 23:03:45.155071    4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af0ac40b526d5b6b4fa4a597572ff9e9d205c384db85782f0828ff46bd2c43b7\": container with ID starting with af0ac40b526d5b6b4fa4a597572ff9e9d205c384db85782f0828ff46bd2c43b7 not found: ID does not exist" containerID="af0ac40b526d5b6b4fa4a597572ff9e9d205c384db85782f0828ff46bd2c43b7"
Oct 30 23:03:45.155160 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:45.155107    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af0ac40b526d5b6b4fa4a597572ff9e9d205c384db85782f0828ff46bd2c43b7"} err="failed to get container status \"af0ac40b526d5b6b4fa4a597572ff9e9d205c384db85782f0828ff46bd2c43b7\": rpc error: code = NotFound desc = could not find container \"af0ac40b526d5b6b4fa4a597572ff9e9d205c384db85782f0828ff46bd2c43b7\": container with ID starting with af0ac40b526d5b6b4fa4a597572ff9e9d205c384db85782f0828ff46bd2c43b7 not found: ID does not exist"
Oct 30 23:03:45.204902 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:45.204872    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5983ea-586646597c-z222v"]
Oct 30 23:03:45.208762 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:45.208736    4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-5983ea-586646597c-z222v"]
Oct 30 23:03:46.192918 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:46.192888    4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="738731d9-e739-4c3e-8936-9c4d01f5fc04" path="/var/lib/kubelet/pods/738731d9-e739-4c3e-8936-9c4d01f5fc04/volumes"
Oct 30 23:03:51.054502 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:51.054462    4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-54114e/premium-w5-54114e-kserve-57f6747bcc-7wjrw" podUID="61990d22-ab8f-45bc-a4d6-4c8d528742f1" containerName="main" containerID="cri-o://823ec907da0c45ca9ffd6f1b97c7e7422a4e68fb5c62fdb6892da92cc7454e65" gracePeriod=45
Oct 30 23:03:51.193373 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:51.193343    4477 generic.go:356] "Generic (PLEG): container finished" podID="61990d22-ab8f-45bc-a4d6-4c8d528742f1" containerID="823ec907da0c45ca9ffd6f1b97c7e7422a4e68fb5c62fdb6892da92cc7454e65" exitCode=0
Oct 30 23:03:51.193497 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:51.193388    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-54114e/premium-w5-54114e-kserve-57f6747bcc-7wjrw" event={"ID":"61990d22-ab8f-45bc-a4d6-4c8d528742f1","Type":"ContainerDied","Data":"823ec907da0c45ca9ffd6f1b97c7e7422a4e68fb5c62fdb6892da92cc7454e65"}
Oct 30 23:03:51.319018 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:51.318957    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-54114e/premium-w5-54114e-kserve-57f6747bcc-7wjrw"
Oct 30 23:03:51.382075 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:51.382023    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61990d22-ab8f-45bc-a4d6-4c8d528742f1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61990d22-ab8f-45bc-a4d6-4c8d528742f1-tmp-dir\") pod \"61990d22-ab8f-45bc-a4d6-4c8d528742f1\" (UID: \"61990d22-ab8f-45bc-a4d6-4c8d528742f1\") "
Oct 30 23:03:51.382233 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:51.382136    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61990d22-ab8f-45bc-a4d6-4c8d528742f1-tls-certs\" (UniqueName: \"kubernetes.io/secret/61990d22-ab8f-45bc-a4d6-4c8d528742f1-tls-certs\") pod \"61990d22-ab8f-45bc-a4d6-4c8d528742f1\" (UID: \"61990d22-ab8f-45bc-a4d6-4c8d528742f1\") "
Oct 30 23:03:51.382303 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:51.382250    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61990d22-ab8f-45bc-a4d6-4c8d528742f1-tmp-dir" pod "61990d22-ab8f-45bc-a4d6-4c8d528742f1" (UID: "61990d22-ab8f-45bc-a4d6-4c8d528742f1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:03:51.382303 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:51.382274    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61990d22-ab8f-45bc-a4d6-4c8d528742f1-home\" (UniqueName: \"kubernetes.io/empty-dir/61990d22-ab8f-45bc-a4d6-4c8d528742f1-home\") pod \"61990d22-ab8f-45bc-a4d6-4c8d528742f1\" (UID: \"61990d22-ab8f-45bc-a4d6-4c8d528742f1\") "
Oct 30 23:03:51.382423 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:51.382315    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61990d22-ab8f-45bc-a4d6-4c8d528742f1-kube-api-access-j8tk4\" (UniqueName: \"kubernetes.io/projected/61990d22-ab8f-45bc-a4d6-4c8d528742f1-kube-api-access-j8tk4\") pod \"61990d22-ab8f-45bc-a4d6-4c8d528742f1\" (UID: \"61990d22-ab8f-45bc-a4d6-4c8d528742f1\") "
Oct 30 23:03:51.382423 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:51.382347    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61990d22-ab8f-45bc-a4d6-4c8d528742f1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61990d22-ab8f-45bc-a4d6-4c8d528742f1-model-cache\") pod \"61990d22-ab8f-45bc-a4d6-4c8d528742f1\" (UID: \"61990d22-ab8f-45bc-a4d6-4c8d528742f1\") "
Oct 30 23:03:51.382423 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:51.382387    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61990d22-ab8f-45bc-a4d6-4c8d528742f1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/61990d22-ab8f-45bc-a4d6-4c8d528742f1-dshm\") pod \"61990d22-ab8f-45bc-a4d6-4c8d528742f1\" (UID: \"61990d22-ab8f-45bc-a4d6-4c8d528742f1\") "
Oct 30 23:03:51.382593 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:51.382426    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61990d22-ab8f-45bc-a4d6-4c8d528742f1-home" pod "61990d22-ab8f-45bc-a4d6-4c8d528742f1" (UID: "61990d22-ab8f-45bc-a4d6-4c8d528742f1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:03:51.382811 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:51.382787    4477 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61990d22-ab8f-45bc-a4d6-4c8d528742f1-tmp-dir\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:03:51.382980 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:51.382967    4477 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61990d22-ab8f-45bc-a4d6-4c8d528742f1-home\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:03:51.383114 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:51.382919    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61990d22-ab8f-45bc-a4d6-4c8d528742f1-model-cache" pod "61990d22-ab8f-45bc-a4d6-4c8d528742f1" (UID: "61990d22-ab8f-45bc-a4d6-4c8d528742f1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:03:51.384558 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:51.384531    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61990d22-ab8f-45bc-a4d6-4c8d528742f1-tls-certs" pod "61990d22-ab8f-45bc-a4d6-4c8d528742f1" (UID: "61990d22-ab8f-45bc-a4d6-4c8d528742f1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:03:51.384967 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:51.384946    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61990d22-ab8f-45bc-a4d6-4c8d528742f1-dshm" pod "61990d22-ab8f-45bc-a4d6-4c8d528742f1" (UID: "61990d22-ab8f-45bc-a4d6-4c8d528742f1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:03:51.388185 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:51.388162    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61990d22-ab8f-45bc-a4d6-4c8d528742f1-kube-api-access-j8tk4" pod "61990d22-ab8f-45bc-a4d6-4c8d528742f1" (UID: "61990d22-ab8f-45bc-a4d6-4c8d528742f1"). InnerVolumeSpecName "kube-api-access-j8tk4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:03:51.484084 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:51.484028    4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8tk4\" (UniqueName: \"kubernetes.io/projected/61990d22-ab8f-45bc-a4d6-4c8d528742f1-kube-api-access-j8tk4\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:03:51.484084 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:51.484084    4477 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61990d22-ab8f-45bc-a4d6-4c8d528742f1-model-cache\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:03:51.484291 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:51.484100    4477 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61990d22-ab8f-45bc-a4d6-4c8d528742f1-dshm\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:03:51.484291 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:51.484112    4477 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61990d22-ab8f-45bc-a4d6-4c8d528742f1-tls-certs\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:03:52.199684 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:52.199653    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-54114e/premium-w5-54114e-kserve-57f6747bcc-7wjrw"
Oct 30 23:03:52.200111 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:52.199783    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-54114e/premium-w5-54114e-kserve-57f6747bcc-7wjrw" event={"ID":"61990d22-ab8f-45bc-a4d6-4c8d528742f1","Type":"ContainerDied","Data":"4dc4c217a29ce4f46a614ba265f7244b75cb30cff11eadbd39b1944a6075f67a"}
Oct 30 23:03:52.200111 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:52.199835    4477 scope.go:122] "RemoveContainer" containerID="823ec907da0c45ca9ffd6f1b97c7e7422a4e68fb5c62fdb6892da92cc7454e65"
Oct 30 23:03:52.246014 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:52.245984    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-54114e/premium-w5-54114e-kserve-57f6747bcc-7wjrw"]
Oct 30 23:03:52.256412 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:52.256389    4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-54114e/premium-w5-54114e-kserve-57f6747bcc-7wjrw"]
Oct 30 23:03:54.193178 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:03:54.193143    4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61990d22-ab8f-45bc-a4d6-4c8d528742f1" path="/var/lib/kubelet/pods/61990d22-ab8f-45bc-a4d6-4c8d528742f1/volumes"
Oct 30 23:03:56.266300 ip-10-0-130-77 kubenswrapper[4477]: E1030 23:03:56.266262    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 23:04:07.043461 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:07.043420    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3cd77f-75497bfcfc-6sdhc"]
Oct 30 23:04:07.043965 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:07.043685    4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3cd77f-75497bfcfc-6sdhc" podUID="0ec852a7-f59e-4475-817a-fb57174a901d" containerName="maas-api" containerID="cri-o://a75ca9b111ba4ef20bde65bf1888b689cf49aff6449ae82964a780ed70bf741d" gracePeriod=30
Oct 30 23:04:07.288574 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:07.288540    4477 generic.go:356] "Generic (PLEG): container finished" podID="0ec852a7-f59e-4475-817a-fb57174a901d" containerID="a75ca9b111ba4ef20bde65bf1888b689cf49aff6449ae82964a780ed70bf741d" exitCode=0
Oct 30 23:04:07.288744 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:07.288577    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3cd77f-75497bfcfc-6sdhc" event={"ID":"0ec852a7-f59e-4475-817a-fb57174a901d","Type":"ContainerDied","Data":"a75ca9b111ba4ef20bde65bf1888b689cf49aff6449ae82964a780ed70bf741d"}
Oct 30 23:04:07.399219 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:07.399195    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3cd77f-75497bfcfc-6sdhc"
Oct 30 23:04:07.419471 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:07.419443    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ec852a7-f59e-4475-817a-fb57174a901d-kube-api-access-qjvk9\" (UniqueName: \"kubernetes.io/projected/0ec852a7-f59e-4475-817a-fb57174a901d-kube-api-access-qjvk9\") pod \"0ec852a7-f59e-4475-817a-fb57174a901d\" (UID: \"0ec852a7-f59e-4475-817a-fb57174a901d\") "
Oct 30 23:04:07.419593 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:07.419517    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ec852a7-f59e-4475-817a-fb57174a901d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ec852a7-f59e-4475-817a-fb57174a901d-maas-api-tls\") pod \"0ec852a7-f59e-4475-817a-fb57174a901d\" (UID: \"0ec852a7-f59e-4475-817a-fb57174a901d\") "
Oct 30 23:04:07.419640 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:07.419601    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ec852a7-f59e-4475-817a-fb57174a901d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ec852a7-f59e-4475-817a-fb57174a901d-metrics-certs\") pod \"0ec852a7-f59e-4475-817a-fb57174a901d\" (UID: \"0ec852a7-f59e-4475-817a-fb57174a901d\") "
Oct 30 23:04:07.419710 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:07.419693    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ec852a7-f59e-4475-817a-fb57174a901d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ec852a7-f59e-4475-817a-fb57174a901d-openshift-service-ca\") pod \"0ec852a7-f59e-4475-817a-fb57174a901d\" (UID: \"0ec852a7-f59e-4475-817a-fb57174a901d\") "
Oct 30 23:04:07.420282 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:07.420236    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ec852a7-f59e-4475-817a-fb57174a901d-openshift-service-ca" pod "0ec852a7-f59e-4475-817a-fb57174a901d" (UID: "0ec852a7-f59e-4475-817a-fb57174a901d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:04:07.422269 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:07.422244    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ec852a7-f59e-4475-817a-fb57174a901d-maas-api-tls" pod "0ec852a7-f59e-4475-817a-fb57174a901d" (UID: "0ec852a7-f59e-4475-817a-fb57174a901d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:04:07.422269 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:07.422262    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ec852a7-f59e-4475-817a-fb57174a901d-kube-api-access-qjvk9" pod "0ec852a7-f59e-4475-817a-fb57174a901d" (UID: "0ec852a7-f59e-4475-817a-fb57174a901d"). InnerVolumeSpecName "kube-api-access-qjvk9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:04:07.422435 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:07.422393    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ec852a7-f59e-4475-817a-fb57174a901d-metrics-certs" pod "0ec852a7-f59e-4475-817a-fb57174a901d" (UID: "0ec852a7-f59e-4475-817a-fb57174a901d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:04:07.520944 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:07.520911    4477 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ec852a7-f59e-4475-817a-fb57174a901d-openshift-service-ca\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:04:07.520944 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:07.520937    4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjvk9\" (UniqueName: \"kubernetes.io/projected/0ec852a7-f59e-4475-817a-fb57174a901d-kube-api-access-qjvk9\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:04:07.520944 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:07.520946    4477 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ec852a7-f59e-4475-817a-fb57174a901d-maas-api-tls\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:04:07.521207 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:07.520955    4477 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0ec852a7-f59e-4475-817a-fb57174a901d-metrics-certs\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:04:08.029845 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:08.029811    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6bz92"]
Oct 30 23:04:08.295319 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:08.295246    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3cd77f-75497bfcfc-6sdhc"
Oct 30 23:04:08.295319 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:08.295256    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3cd77f-75497bfcfc-6sdhc" event={"ID":"0ec852a7-f59e-4475-817a-fb57174a901d","Type":"ContainerDied","Data":"4234b978d68784b107ea46428831ae863de84b8c05e1ed84d258b1ea5ead9c8c"}
Oct 30 23:04:08.295826 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:08.295324    4477 scope.go:122] "RemoveContainer" containerID="a75ca9b111ba4ef20bde65bf1888b689cf49aff6449ae82964a780ed70bf741d"
Oct 30 23:04:08.342996 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:08.342969    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3cd77f-75497bfcfc-6sdhc"]
Oct 30 23:04:08.353151 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:08.353124    4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-3cd77f-75497bfcfc-6sdhc"]
Oct 30 23:04:10.193347 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:10.193317    4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ec852a7-f59e-4475-817a-fb57174a901d" path="/var/lib/kubelet/pods/0ec852a7-f59e-4475-817a-fb57174a901d/volumes"
Oct 30 23:04:11.272344 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:11.272298    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-012ba0-798cb567cc-k8959"]
Oct 30 23:04:11.272841 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:11.272524    4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-012ba0-798cb567cc-k8959" podUID="130008a5-8f15-4ded-b8bd-d1903cb436b2" containerName="maas-api" containerID="cri-o://7c3e38a794382bbf21dc51ce6eb57f9c087a5baebdf5300880aed557eb5ca4ec" gracePeriod=30
Oct 30 23:04:11.595070 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:11.594813    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-012ba0-798cb567cc-k8959"
Oct 30 23:04:11.675084 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:11.675025    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/130008a5-8f15-4ded-b8bd-d1903cb436b2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/130008a5-8f15-4ded-b8bd-d1903cb436b2-metrics-certs\") pod \"130008a5-8f15-4ded-b8bd-d1903cb436b2\" (UID: \"130008a5-8f15-4ded-b8bd-d1903cb436b2\") "
Oct 30 23:04:11.675269 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:11.675125    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/130008a5-8f15-4ded-b8bd-d1903cb436b2-kube-api-access-jn86p\" (UniqueName: \"kubernetes.io/projected/130008a5-8f15-4ded-b8bd-d1903cb436b2-kube-api-access-jn86p\") pod \"130008a5-8f15-4ded-b8bd-d1903cb436b2\" (UID: \"130008a5-8f15-4ded-b8bd-d1903cb436b2\") "
Oct 30 23:04:11.675269 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:11.675168    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/130008a5-8f15-4ded-b8bd-d1903cb436b2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/130008a5-8f15-4ded-b8bd-d1903cb436b2-maas-api-tls\") pod \"130008a5-8f15-4ded-b8bd-d1903cb436b2\" (UID: \"130008a5-8f15-4ded-b8bd-d1903cb436b2\") "
Oct 30 23:04:11.675408 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:11.675323    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/130008a5-8f15-4ded-b8bd-d1903cb436b2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/130008a5-8f15-4ded-b8bd-d1903cb436b2-openshift-service-ca\") pod \"130008a5-8f15-4ded-b8bd-d1903cb436b2\" (UID: \"130008a5-8f15-4ded-b8bd-d1903cb436b2\") "
Oct 30 23:04:11.676469 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:11.676405    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/130008a5-8f15-4ded-b8bd-d1903cb436b2-openshift-service-ca" pod "130008a5-8f15-4ded-b8bd-d1903cb436b2" (UID: "130008a5-8f15-4ded-b8bd-d1903cb436b2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:04:11.691921 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:11.691840    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/130008a5-8f15-4ded-b8bd-d1903cb436b2-kube-api-access-jn86p" pod "130008a5-8f15-4ded-b8bd-d1903cb436b2" (UID: "130008a5-8f15-4ded-b8bd-d1903cb436b2"). InnerVolumeSpecName "kube-api-access-jn86p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:04:11.693769 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:11.693737    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/130008a5-8f15-4ded-b8bd-d1903cb436b2-maas-api-tls" pod "130008a5-8f15-4ded-b8bd-d1903cb436b2" (UID: "130008a5-8f15-4ded-b8bd-d1903cb436b2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:04:11.694036 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:11.694007    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/130008a5-8f15-4ded-b8bd-d1903cb436b2-metrics-certs" pod "130008a5-8f15-4ded-b8bd-d1903cb436b2" (UID: "130008a5-8f15-4ded-b8bd-d1903cb436b2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:04:11.776564 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:11.776530    4477 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/130008a5-8f15-4ded-b8bd-d1903cb436b2-metrics-certs\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:04:11.776564 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:11.776567    4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jn86p\" (UniqueName: \"kubernetes.io/projected/130008a5-8f15-4ded-b8bd-d1903cb436b2-kube-api-access-jn86p\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:04:11.776771 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:11.776581    4477 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/130008a5-8f15-4ded-b8bd-d1903cb436b2-maas-api-tls\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:04:11.776771 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:11.776595    4477 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/130008a5-8f15-4ded-b8bd-d1903cb436b2-openshift-service-ca\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:04:12.319111 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:12.319073    4477 generic.go:356] "Generic (PLEG): container finished" podID="130008a5-8f15-4ded-b8bd-d1903cb436b2" containerID="7c3e38a794382bbf21dc51ce6eb57f9c087a5baebdf5300880aed557eb5ca4ec" exitCode=0
Oct 30 23:04:12.319591 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:12.319126    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-012ba0-798cb567cc-k8959"
Oct 30 23:04:12.319591 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:12.319159    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-012ba0-798cb567cc-k8959" event={"ID":"130008a5-8f15-4ded-b8bd-d1903cb436b2","Type":"ContainerDied","Data":"7c3e38a794382bbf21dc51ce6eb57f9c087a5baebdf5300880aed557eb5ca4ec"}
Oct 30 23:04:12.319591 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:12.319215    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-012ba0-798cb567cc-k8959" event={"ID":"130008a5-8f15-4ded-b8bd-d1903cb436b2","Type":"ContainerDied","Data":"e9f8871af29e9291c6f000a10ce55ee482bc42a79a5fee6d781bc12b49bae425"}
Oct 30 23:04:12.319591 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:12.319238    4477 scope.go:122] "RemoveContainer" containerID="7c3e38a794382bbf21dc51ce6eb57f9c087a5baebdf5300880aed557eb5ca4ec"
Oct 30 23:04:12.328411 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:12.328386    4477 scope.go:122] "RemoveContainer" containerID="7c3e38a794382bbf21dc51ce6eb57f9c087a5baebdf5300880aed557eb5ca4ec"
Oct 30 23:04:12.328675 ip-10-0-130-77 kubenswrapper[4477]: E1030 23:04:12.328658    4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c3e38a794382bbf21dc51ce6eb57f9c087a5baebdf5300880aed557eb5ca4ec\": container with ID starting with 7c3e38a794382bbf21dc51ce6eb57f9c087a5baebdf5300880aed557eb5ca4ec not found: ID does not exist" containerID="7c3e38a794382bbf21dc51ce6eb57f9c087a5baebdf5300880aed557eb5ca4ec"
Oct 30 23:04:12.328729 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:12.328684    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c3e38a794382bbf21dc51ce6eb57f9c087a5baebdf5300880aed557eb5ca4ec"} err="failed to get container status \"7c3e38a794382bbf21dc51ce6eb57f9c087a5baebdf5300880aed557eb5ca4ec\": rpc error: code = NotFound desc = could not find container \"7c3e38a794382bbf21dc51ce6eb57f9c087a5baebdf5300880aed557eb5ca4ec\": container with ID starting with 7c3e38a794382bbf21dc51ce6eb57f9c087a5baebdf5300880aed557eb5ca4ec not found: ID does not exist"
Oct 30 23:04:12.373793 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:12.373767    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-012ba0-798cb567cc-k8959"]
Oct 30 23:04:12.376586 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:12.376562    4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-012ba0-798cb567cc-k8959"]
Oct 30 23:04:14.193597 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:14.193563    4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="130008a5-8f15-4ded-b8bd-d1903cb436b2" path="/var/lib/kubelet/pods/130008a5-8f15-4ded-b8bd-d1903cb436b2/volumes"
Oct 30 23:04:15.862467 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:15.862433    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-23aab2-55d6dff57-kp9bb"]
Oct 30 23:04:15.863167 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:15.863131    4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-23aab2-55d6dff57-kp9bb" podUID="d12a3703-b587-4f7b-9769-d4322c840eeb" containerName="maas-api" containerID="cri-o://7e663b22a0e1326e4ac321d868a9e74d178016efe5d4cdd09d3b36f048414148" gracePeriod=30
Oct 30 23:04:16.207130 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:16.207100    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-23aab2-55d6dff57-kp9bb"
Oct 30 23:04:16.296088 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:16.274115    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6af86e-57878656ff-qv65k"]
Oct 30 23:04:16.296088 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:16.274429    4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6af86e-57878656ff-qv65k" podUID="763c695b-9f27-48e4-80a6-ae3c22bc9816" containerName="maas-api" containerID="cri-o://0c0de0b66292e3ad1c60c817316d1f69abd7006fa228e14233610401b441faf4" gracePeriod=30
Oct 30 23:04:16.360643 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:16.359478    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d12a3703-b587-4f7b-9769-d4322c840eeb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d12a3703-b587-4f7b-9769-d4322c840eeb-maas-api-tls\") pod \"d12a3703-b587-4f7b-9769-d4322c840eeb\" (UID: \"d12a3703-b587-4f7b-9769-d4322c840eeb\") "
Oct 30 23:04:16.360643 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:16.359563    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d12a3703-b587-4f7b-9769-d4322c840eeb-kube-api-access-kflw7\" (UniqueName: \"kubernetes.io/projected/d12a3703-b587-4f7b-9769-d4322c840eeb-kube-api-access-kflw7\") pod \"d12a3703-b587-4f7b-9769-d4322c840eeb\" (UID: \"d12a3703-b587-4f7b-9769-d4322c840eeb\") "
Oct 30 23:04:16.360643 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:16.359593    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d12a3703-b587-4f7b-9769-d4322c840eeb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d12a3703-b587-4f7b-9769-d4322c840eeb-metrics-certs\") pod \"d12a3703-b587-4f7b-9769-d4322c840eeb\" (UID: \"d12a3703-b587-4f7b-9769-d4322c840eeb\") "
Oct 30 23:04:16.360643 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:16.359681    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d12a3703-b587-4f7b-9769-d4322c840eeb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d12a3703-b587-4f7b-9769-d4322c840eeb-openshift-service-ca\") pod \"d12a3703-b587-4f7b-9769-d4322c840eeb\" (UID: \"d12a3703-b587-4f7b-9769-d4322c840eeb\") "
Oct 30 23:04:16.361416 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:16.361145    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d12a3703-b587-4f7b-9769-d4322c840eeb-openshift-service-ca" pod "d12a3703-b587-4f7b-9769-d4322c840eeb" (UID: "d12a3703-b587-4f7b-9769-d4322c840eeb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:04:16.363338 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:16.363310    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d12a3703-b587-4f7b-9769-d4322c840eeb-kube-api-access-kflw7" pod "d12a3703-b587-4f7b-9769-d4322c840eeb" (UID: "d12a3703-b587-4f7b-9769-d4322c840eeb"). InnerVolumeSpecName "kube-api-access-kflw7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:04:16.363623 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:16.363591    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d12a3703-b587-4f7b-9769-d4322c840eeb-maas-api-tls" pod "d12a3703-b587-4f7b-9769-d4322c840eeb" (UID: "d12a3703-b587-4f7b-9769-d4322c840eeb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:04:16.364846 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:16.364819    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d12a3703-b587-4f7b-9769-d4322c840eeb-metrics-certs" pod "d12a3703-b587-4f7b-9769-d4322c840eeb" (UID: "d12a3703-b587-4f7b-9769-d4322c840eeb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:04:16.367036 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:16.366401    4477 generic.go:356] "Generic (PLEG): container finished" podID="d12a3703-b587-4f7b-9769-d4322c840eeb" containerID="7e663b22a0e1326e4ac321d868a9e74d178016efe5d4cdd09d3b36f048414148" exitCode=0
Oct 30 23:04:16.367036 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:16.366445    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-23aab2-55d6dff57-kp9bb"
Oct 30 23:04:16.367036 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:16.366550    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-23aab2-55d6dff57-kp9bb" event={"ID":"d12a3703-b587-4f7b-9769-d4322c840eeb","Type":"ContainerDied","Data":"7e663b22a0e1326e4ac321d868a9e74d178016efe5d4cdd09d3b36f048414148"}
Oct 30 23:04:16.367036 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:16.366589    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-23aab2-55d6dff57-kp9bb" event={"ID":"d12a3703-b587-4f7b-9769-d4322c840eeb","Type":"ContainerDied","Data":"8ff918e43f5c229eb8b66cbd1de36ff13794b840e679f57191d9af4b4212b5e7"}
Oct 30 23:04:16.367036 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:16.366614    4477 scope.go:122] "RemoveContainer" containerID="7e663b22a0e1326e4ac321d868a9e74d178016efe5d4cdd09d3b36f048414148"
Oct 30 23:04:16.432841 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:16.432815    4477 scope.go:122] "RemoveContainer" containerID="7e663b22a0e1326e4ac321d868a9e74d178016efe5d4cdd09d3b36f048414148"
Oct 30 23:04:16.433300 ip-10-0-130-77 kubenswrapper[4477]: E1030 23:04:16.433223    4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e663b22a0e1326e4ac321d868a9e74d178016efe5d4cdd09d3b36f048414148\": container with ID starting with 7e663b22a0e1326e4ac321d868a9e74d178016efe5d4cdd09d3b36f048414148 not found: ID does not exist" containerID="7e663b22a0e1326e4ac321d868a9e74d178016efe5d4cdd09d3b36f048414148"
Oct 30 23:04:16.433300 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:16.433269    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e663b22a0e1326e4ac321d868a9e74d178016efe5d4cdd09d3b36f048414148"} err="failed to get container status \"7e663b22a0e1326e4ac321d868a9e74d178016efe5d4cdd09d3b36f048414148\": rpc error: code = NotFound desc = could not find container \"7e663b22a0e1326e4ac321d868a9e74d178016efe5d4cdd09d3b36f048414148\": container with ID starting with 7e663b22a0e1326e4ac321d868a9e74d178016efe5d4cdd09d3b36f048414148 not found: ID does not exist"
Oct 30 23:04:16.441385 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:16.441359    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-23aab2-55d6dff57-kp9bb"]
Oct 30 23:04:16.455617 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:16.455578    4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-23aab2-55d6dff57-kp9bb"]
Oct 30 23:04:16.470902 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:16.470868    4477 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d12a3703-b587-4f7b-9769-d4322c840eeb-maas-api-tls\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:04:16.475472 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:16.475449    4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kflw7\" (UniqueName: \"kubernetes.io/projected/d12a3703-b587-4f7b-9769-d4322c840eeb-kube-api-access-kflw7\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:04:16.475679 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:16.475662    4477 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d12a3703-b587-4f7b-9769-d4322c840eeb-metrics-certs\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:04:16.475802 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:16.475789    4477 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d12a3703-b587-4f7b-9769-d4322c840eeb-openshift-service-ca\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:04:16.617501 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:16.617472    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6af86e-57878656ff-qv65k"
Oct 30 23:04:16.784564 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:16.784495    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/763c695b-9f27-48e4-80a6-ae3c22bc9816-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/763c695b-9f27-48e4-80a6-ae3c22bc9816-openshift-service-ca\") pod \"763c695b-9f27-48e4-80a6-ae3c22bc9816\" (UID: \"763c695b-9f27-48e4-80a6-ae3c22bc9816\") "
Oct 30 23:04:16.784753 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:16.784577    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/763c695b-9f27-48e4-80a6-ae3c22bc9816-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/763c695b-9f27-48e4-80a6-ae3c22bc9816-maas-api-tls\") pod \"763c695b-9f27-48e4-80a6-ae3c22bc9816\" (UID: \"763c695b-9f27-48e4-80a6-ae3c22bc9816\") "
Oct 30 23:04:16.784753 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:16.784692    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/763c695b-9f27-48e4-80a6-ae3c22bc9816-kube-api-access-qp2pm\" (UniqueName: \"kubernetes.io/projected/763c695b-9f27-48e4-80a6-ae3c22bc9816-kube-api-access-qp2pm\") pod \"763c695b-9f27-48e4-80a6-ae3c22bc9816\" (UID: \"763c695b-9f27-48e4-80a6-ae3c22bc9816\") "
Oct 30 23:04:16.784875 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:16.784809    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/763c695b-9f27-48e4-80a6-ae3c22bc9816-metrics-certs\" (UniqueName: \"kubernetes.io/secret/763c695b-9f27-48e4-80a6-ae3c22bc9816-metrics-certs\") pod \"763c695b-9f27-48e4-80a6-ae3c22bc9816\" (UID: \"763c695b-9f27-48e4-80a6-ae3c22bc9816\") "
Oct 30 23:04:16.784975 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:16.784950    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/763c695b-9f27-48e4-80a6-ae3c22bc9816-openshift-service-ca" pod "763c695b-9f27-48e4-80a6-ae3c22bc9816" (UID: "763c695b-9f27-48e4-80a6-ae3c22bc9816"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:04:16.785440 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:16.785395    4477 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/763c695b-9f27-48e4-80a6-ae3c22bc9816-openshift-service-ca\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:04:16.788397 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:16.788359    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/763c695b-9f27-48e4-80a6-ae3c22bc9816-kube-api-access-qp2pm" pod "763c695b-9f27-48e4-80a6-ae3c22bc9816" (UID: "763c695b-9f27-48e4-80a6-ae3c22bc9816"). InnerVolumeSpecName "kube-api-access-qp2pm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:04:16.790445 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:16.790424    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/763c695b-9f27-48e4-80a6-ae3c22bc9816-metrics-certs" pod "763c695b-9f27-48e4-80a6-ae3c22bc9816" (UID: "763c695b-9f27-48e4-80a6-ae3c22bc9816"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:04:16.790545 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:16.790454    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/763c695b-9f27-48e4-80a6-ae3c22bc9816-maas-api-tls" pod "763c695b-9f27-48e4-80a6-ae3c22bc9816" (UID: "763c695b-9f27-48e4-80a6-ae3c22bc9816"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:04:16.886794 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:16.886770    4477 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/763c695b-9f27-48e4-80a6-ae3c22bc9816-maas-api-tls\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:04:16.886794 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:16.886793    4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qp2pm\" (UniqueName: \"kubernetes.io/projected/763c695b-9f27-48e4-80a6-ae3c22bc9816-kube-api-access-qp2pm\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:04:16.887159 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:16.886803    4477 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/763c695b-9f27-48e4-80a6-ae3c22bc9816-metrics-certs\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:04:17.386537 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:17.384500    4477 generic.go:356] "Generic (PLEG): container finished" podID="763c695b-9f27-48e4-80a6-ae3c22bc9816" containerID="0c0de0b66292e3ad1c60c817316d1f69abd7006fa228e14233610401b441faf4" exitCode=0
Oct 30 23:04:17.386537 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:17.384605    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6af86e-57878656ff-qv65k" event={"ID":"763c695b-9f27-48e4-80a6-ae3c22bc9816","Type":"ContainerDied","Data":"0c0de0b66292e3ad1c60c817316d1f69abd7006fa228e14233610401b441faf4"}
Oct 30 23:04:17.386537 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:17.384690    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6af86e-57878656ff-qv65k" event={"ID":"763c695b-9f27-48e4-80a6-ae3c22bc9816","Type":"ContainerDied","Data":"600eedb45dd6bed6ad95a553f79e0f063bc1a7d14c40dc659d8a2912a8c7847d"}
Oct 30 23:04:17.386537 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:17.384718    4477 scope.go:122] "RemoveContainer" containerID="0c0de0b66292e3ad1c60c817316d1f69abd7006fa228e14233610401b441faf4"
Oct 30 23:04:17.386537 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:17.384864    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6af86e-57878656ff-qv65k"
Oct 30 23:04:17.397609 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:17.397589    4477 scope.go:122] "RemoveContainer" containerID="0c0de0b66292e3ad1c60c817316d1f69abd7006fa228e14233610401b441faf4"
Oct 30 23:04:17.397894 ip-10-0-130-77 kubenswrapper[4477]: E1030 23:04:17.397865    4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c0de0b66292e3ad1c60c817316d1f69abd7006fa228e14233610401b441faf4\": container with ID starting with 0c0de0b66292e3ad1c60c817316d1f69abd7006fa228e14233610401b441faf4 not found: ID does not exist" containerID="0c0de0b66292e3ad1c60c817316d1f69abd7006fa228e14233610401b441faf4"
Oct 30 23:04:17.397945 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:17.397906    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c0de0b66292e3ad1c60c817316d1f69abd7006fa228e14233610401b441faf4"} err="failed to get container status \"0c0de0b66292e3ad1c60c817316d1f69abd7006fa228e14233610401b441faf4\": rpc error: code = NotFound desc = could not find container \"0c0de0b66292e3ad1c60c817316d1f69abd7006fa228e14233610401b441faf4\": container with ID starting with 0c0de0b66292e3ad1c60c817316d1f69abd7006fa228e14233610401b441faf4 not found: ID does not exist"
Oct 30 23:04:17.460085 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:17.460055    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6af86e-57878656ff-qv65k"]
Oct 30 23:04:17.462939 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:17.462913    4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-6af86e-57878656ff-qv65k"]
Oct 30 23:04:18.201533 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:18.201491    4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="763c695b-9f27-48e4-80a6-ae3c22bc9816" path="/var/lib/kubelet/pods/763c695b-9f27-48e4-80a6-ae3c22bc9816/volumes"
Oct 30 23:04:18.202224 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:18.202137    4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d12a3703-b587-4f7b-9769-d4322c840eeb" path="/var/lib/kubelet/pods/d12a3703-b587-4f7b-9769-d4322c840eeb/volumes"
Oct 30 23:04:21.532431 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:21.532391    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-421583/distinct-w2-421583-kserve-69cf5f4d6b-qbgnt"]
Oct 30 23:04:26.265135 ip-10-0-130-77 kubenswrapper[4477]: E1030 23:04:26.265096    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 23:04:29.352184 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:29.352151    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-2b1e88/simulator-w1-2b1e88-kserve-6c579b99f8-q5wpq"]
Oct 30 23:04:30.857649 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:30.857618    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-09d0d5/distinct-2-w0-09d0d5-kserve-7dcf89b98b-7cjph"]
Oct 30 23:04:35.311367 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:35.311335    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-bff950-c46cf8dc-95t2k"]
Oct 30 23:04:35.311848 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:35.311530    4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bff950-c46cf8dc-95t2k" podUID="a637e6be-489f-4d98-bc02-bca751282d13" containerName="maas-api" containerID="cri-o://54194ed0b25ccae4f9d3a678bf02c6b0899ffb8fa8998b98ecf81b7f0d999560" gracePeriod=30
Oct 30 23:04:35.509185 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:35.508781    4477 generic.go:356] "Generic (PLEG): container finished" podID="a637e6be-489f-4d98-bc02-bca751282d13" containerID="54194ed0b25ccae4f9d3a678bf02c6b0899ffb8fa8998b98ecf81b7f0d999560" exitCode=0
Oct 30 23:04:35.509185 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:35.508976    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bff950-c46cf8dc-95t2k" event={"ID":"a637e6be-489f-4d98-bc02-bca751282d13","Type":"ContainerDied","Data":"54194ed0b25ccae4f9d3a678bf02c6b0899ffb8fa8998b98ecf81b7f0d999560"}
Oct 30 23:04:35.594631 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:35.594602    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bff950-c46cf8dc-95t2k"
Oct 30 23:04:35.728937 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:35.728905    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a637e6be-489f-4d98-bc02-bca751282d13-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a637e6be-489f-4d98-bc02-bca751282d13-metrics-certs\") pod \"a637e6be-489f-4d98-bc02-bca751282d13\" (UID: \"a637e6be-489f-4d98-bc02-bca751282d13\") "
Oct 30 23:04:35.729144 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:35.728964    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a637e6be-489f-4d98-bc02-bca751282d13-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a637e6be-489f-4d98-bc02-bca751282d13-openshift-service-ca\") pod \"a637e6be-489f-4d98-bc02-bca751282d13\" (UID: \"a637e6be-489f-4d98-bc02-bca751282d13\") "
Oct 30 23:04:35.729144 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:35.729071    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a637e6be-489f-4d98-bc02-bca751282d13-kube-api-access-6hvf2\" (UniqueName: \"kubernetes.io/projected/a637e6be-489f-4d98-bc02-bca751282d13-kube-api-access-6hvf2\") pod \"a637e6be-489f-4d98-bc02-bca751282d13\" (UID: \"a637e6be-489f-4d98-bc02-bca751282d13\") "
Oct 30 23:04:35.729290 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:35.729194    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a637e6be-489f-4d98-bc02-bca751282d13-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a637e6be-489f-4d98-bc02-bca751282d13-maas-api-tls\") pod \"a637e6be-489f-4d98-bc02-bca751282d13\" (UID: \"a637e6be-489f-4d98-bc02-bca751282d13\") "
Oct 30 23:04:35.729406 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:35.729369    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a637e6be-489f-4d98-bc02-bca751282d13-openshift-service-ca" pod "a637e6be-489f-4d98-bc02-bca751282d13" (UID: "a637e6be-489f-4d98-bc02-bca751282d13"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:04:35.729761 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:35.729735    4477 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a637e6be-489f-4d98-bc02-bca751282d13-openshift-service-ca\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:04:35.731240 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:35.731212    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a637e6be-489f-4d98-bc02-bca751282d13-kube-api-access-6hvf2" pod "a637e6be-489f-4d98-bc02-bca751282d13" (UID: "a637e6be-489f-4d98-bc02-bca751282d13"). InnerVolumeSpecName "kube-api-access-6hvf2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:04:35.731240 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:35.731228    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a637e6be-489f-4d98-bc02-bca751282d13-metrics-certs" pod "a637e6be-489f-4d98-bc02-bca751282d13" (UID: "a637e6be-489f-4d98-bc02-bca751282d13"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:04:35.731364 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:35.731289    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a637e6be-489f-4d98-bc02-bca751282d13-maas-api-tls" pod "a637e6be-489f-4d98-bc02-bca751282d13" (UID: "a637e6be-489f-4d98-bc02-bca751282d13"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:04:35.837108 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:35.832889    4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hvf2\" (UniqueName: \"kubernetes.io/projected/a637e6be-489f-4d98-bc02-bca751282d13-kube-api-access-6hvf2\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:04:35.837108 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:35.832949    4477 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a637e6be-489f-4d98-bc02-bca751282d13-maas-api-tls\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:04:35.837108 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:35.832967    4477 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a637e6be-489f-4d98-bc02-bca751282d13-metrics-certs\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:04:36.515767 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:36.515687    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bff950-c46cf8dc-95t2k"
Oct 30 23:04:36.515767 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:36.515710    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-bff950-c46cf8dc-95t2k" event={"ID":"a637e6be-489f-4d98-bc02-bca751282d13","Type":"ContainerDied","Data":"fa88e20845a2469fdadc9aa042dca866e725c4f35d2f3538b0d2551c1191e0fa"}
Oct 30 23:04:36.516245 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:36.515797    4477 scope.go:122] "RemoveContainer" containerID="54194ed0b25ccae4f9d3a678bf02c6b0899ffb8fa8998b98ecf81b7f0d999560"
Oct 30 23:04:36.553332 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:36.553302    4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-421583/distinct-w2-421583-kserve-69cf5f4d6b-qbgnt" podUID="510ad10d-0554-47c8-9b55-deb359795071" containerName="main" containerID="cri-o://63dfc3553ebbd4e7044838049dc17fbae884a6d5b708b0a5763a81f774fbb1fd" gracePeriod=45
Oct 30 23:04:36.561621 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:36.561595    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-bff950-c46cf8dc-95t2k"]
Oct 30 23:04:36.571793 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:36.571774    4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-bff950-c46cf8dc-95t2k"]
Oct 30 23:04:36.835733 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:36.835709    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-421583/distinct-w2-421583-kserve-69cf5f4d6b-qbgnt"
Oct 30 23:04:36.954836 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:36.954807    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/510ad10d-0554-47c8-9b55-deb359795071-dshm\" (UniqueName: \"kubernetes.io/empty-dir/510ad10d-0554-47c8-9b55-deb359795071-dshm\") pod \"510ad10d-0554-47c8-9b55-deb359795071\" (UID: \"510ad10d-0554-47c8-9b55-deb359795071\") "
Oct 30 23:04:36.955026 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:36.954846    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/510ad10d-0554-47c8-9b55-deb359795071-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/510ad10d-0554-47c8-9b55-deb359795071-model-cache\") pod \"510ad10d-0554-47c8-9b55-deb359795071\" (UID: \"510ad10d-0554-47c8-9b55-deb359795071\") "
Oct 30 23:04:36.955026 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:36.954880    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/510ad10d-0554-47c8-9b55-deb359795071-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/510ad10d-0554-47c8-9b55-deb359795071-tmp-dir\") pod \"510ad10d-0554-47c8-9b55-deb359795071\" (UID: \"510ad10d-0554-47c8-9b55-deb359795071\") "
Oct 30 23:04:36.955026 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:36.955015    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/510ad10d-0554-47c8-9b55-deb359795071-kube-api-access-flh8t\" (UniqueName: \"kubernetes.io/projected/510ad10d-0554-47c8-9b55-deb359795071-kube-api-access-flh8t\") pod \"510ad10d-0554-47c8-9b55-deb359795071\" (UID: \"510ad10d-0554-47c8-9b55-deb359795071\") "
Oct 30 23:04:36.955245 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:36.955033    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/510ad10d-0554-47c8-9b55-deb359795071-tls-certs\" (UniqueName: \"kubernetes.io/secret/510ad10d-0554-47c8-9b55-deb359795071-tls-certs\") pod \"510ad10d-0554-47c8-9b55-deb359795071\" (UID: \"510ad10d-0554-47c8-9b55-deb359795071\") "
Oct 30 23:04:36.955245 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:36.955061    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/510ad10d-0554-47c8-9b55-deb359795071-home\" (UniqueName: \"kubernetes.io/empty-dir/510ad10d-0554-47c8-9b55-deb359795071-home\") pod \"510ad10d-0554-47c8-9b55-deb359795071\" (UID: \"510ad10d-0554-47c8-9b55-deb359795071\") "
Oct 30 23:04:36.955245 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:36.955159    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/510ad10d-0554-47c8-9b55-deb359795071-model-cache" pod "510ad10d-0554-47c8-9b55-deb359795071" (UID: "510ad10d-0554-47c8-9b55-deb359795071"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:04:36.955245 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:36.955183    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/510ad10d-0554-47c8-9b55-deb359795071-tmp-dir" pod "510ad10d-0554-47c8-9b55-deb359795071" (UID: "510ad10d-0554-47c8-9b55-deb359795071"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:04:36.955409 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:36.955382    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/510ad10d-0554-47c8-9b55-deb359795071-home" pod "510ad10d-0554-47c8-9b55-deb359795071" (UID: "510ad10d-0554-47c8-9b55-deb359795071"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:04:36.955538 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:36.955518    4477 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/510ad10d-0554-47c8-9b55-deb359795071-home\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:04:36.955575 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:36.955545    4477 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/510ad10d-0554-47c8-9b55-deb359795071-model-cache\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:04:36.955575 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:36.955563    4477 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/510ad10d-0554-47c8-9b55-deb359795071-tmp-dir\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:04:36.957277 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:36.957246    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/510ad10d-0554-47c8-9b55-deb359795071-tls-certs" pod "510ad10d-0554-47c8-9b55-deb359795071" (UID: "510ad10d-0554-47c8-9b55-deb359795071"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:04:36.957277 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:36.957260    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/510ad10d-0554-47c8-9b55-deb359795071-dshm" pod "510ad10d-0554-47c8-9b55-deb359795071" (UID: "510ad10d-0554-47c8-9b55-deb359795071"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:04:36.957277 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:36.957269    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/510ad10d-0554-47c8-9b55-deb359795071-kube-api-access-flh8t" pod "510ad10d-0554-47c8-9b55-deb359795071" (UID: "510ad10d-0554-47c8-9b55-deb359795071"). InnerVolumeSpecName "kube-api-access-flh8t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:04:37.056741 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:37.056708    4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flh8t\" (UniqueName: \"kubernetes.io/projected/510ad10d-0554-47c8-9b55-deb359795071-kube-api-access-flh8t\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:04:37.056741 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:37.056738    4477 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/510ad10d-0554-47c8-9b55-deb359795071-tls-certs\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:04:37.056741 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:37.056748    4477 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/510ad10d-0554-47c8-9b55-deb359795071-dshm\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:04:37.522629 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:37.522588    4477 generic.go:356] "Generic (PLEG): container finished" podID="510ad10d-0554-47c8-9b55-deb359795071" containerID="63dfc3553ebbd4e7044838049dc17fbae884a6d5b708b0a5763a81f774fbb1fd" exitCode=0
Oct 30 23:04:37.523315 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:37.522631    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-421583/distinct-w2-421583-kserve-69cf5f4d6b-qbgnt"
Oct 30 23:04:37.523315 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:37.522668    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-421583/distinct-w2-421583-kserve-69cf5f4d6b-qbgnt" event={"ID":"510ad10d-0554-47c8-9b55-deb359795071","Type":"ContainerDied","Data":"63dfc3553ebbd4e7044838049dc17fbae884a6d5b708b0a5763a81f774fbb1fd"}
Oct 30 23:04:37.523315 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:37.522716    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-421583/distinct-w2-421583-kserve-69cf5f4d6b-qbgnt" event={"ID":"510ad10d-0554-47c8-9b55-deb359795071","Type":"ContainerDied","Data":"3404620da8b739cff56a3be61466701871d2400bd57f2d82fc434a48e35134ca"}
Oct 30 23:04:37.523315 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:37.522744    4477 scope.go:122] "RemoveContainer" containerID="63dfc3553ebbd4e7044838049dc17fbae884a6d5b708b0a5763a81f774fbb1fd"
Oct 30 23:04:37.532528 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:37.532513    4477 scope.go:122] "RemoveContainer" containerID="63dfc3553ebbd4e7044838049dc17fbae884a6d5b708b0a5763a81f774fbb1fd"
Oct 30 23:04:37.532808 ip-10-0-130-77 kubenswrapper[4477]: E1030 23:04:37.532791    4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63dfc3553ebbd4e7044838049dc17fbae884a6d5b708b0a5763a81f774fbb1fd\": container with ID starting with 63dfc3553ebbd4e7044838049dc17fbae884a6d5b708b0a5763a81f774fbb1fd not found: ID does not exist" containerID="63dfc3553ebbd4e7044838049dc17fbae884a6d5b708b0a5763a81f774fbb1fd"
Oct 30 23:04:37.532864 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:37.532815    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63dfc3553ebbd4e7044838049dc17fbae884a6d5b708b0a5763a81f774fbb1fd"} err="failed to get container status \"63dfc3553ebbd4e7044838049dc17fbae884a6d5b708b0a5763a81f774fbb1fd\": rpc error: code = NotFound desc = could not find container \"63dfc3553ebbd4e7044838049dc17fbae884a6d5b708b0a5763a81f774fbb1fd\": container with ID starting with 63dfc3553ebbd4e7044838049dc17fbae884a6d5b708b0a5763a81f774fbb1fd not found: ID does not exist"
Oct 30 23:04:37.568381 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:37.568354    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-421583/distinct-w2-421583-kserve-69cf5f4d6b-qbgnt"]
Oct 30 23:04:37.578239 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:37.578219    4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-421583/distinct-w2-421583-kserve-69cf5f4d6b-qbgnt"]
Oct 30 23:04:38.193892 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:38.193864    4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="510ad10d-0554-47c8-9b55-deb359795071" path="/var/lib/kubelet/pods/510ad10d-0554-47c8-9b55-deb359795071/volumes"
Oct 30 23:04:38.194359 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:38.194343    4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a637e6be-489f-4d98-bc02-bca751282d13" path="/var/lib/kubelet/pods/a637e6be-489f-4d98-bc02-bca751282d13/volumes"
Oct 30 23:04:44.373244 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:44.373149    4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-2b1e88/simulator-w1-2b1e88-kserve-6c579b99f8-q5wpq" podUID="722f10d8-b7a2-4bf7-a274-9669529ef427" containerName="main" containerID="cri-o://bafc7cd2e69bc49eba568e173fdf9c3499aefad1893df48c60d0ed73a7600ffb" gracePeriod=45
Oct 30 23:04:44.443646 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:44.443603    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-2b1e88/simulator-w1-2b1e88-kserve-6c579b99f8-q5wpq" podUID="722f10d8-b7a2-4bf7-a274-9669529ef427" containerName="main" probeResult="failure" output="Get \"https://10.134.0.43:8000/ready\": dial tcp 10.134.0.43:8000: connect: connection refused"
Oct 30 23:04:44.577220 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:44.577187    4477 generic.go:356] "Generic (PLEG): container finished" podID="722f10d8-b7a2-4bf7-a274-9669529ef427" containerID="bafc7cd2e69bc49eba568e173fdf9c3499aefad1893df48c60d0ed73a7600ffb" exitCode=0
Oct 30 23:04:44.577356 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:44.577267    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-2b1e88/simulator-w1-2b1e88-kserve-6c579b99f8-q5wpq" event={"ID":"722f10d8-b7a2-4bf7-a274-9669529ef427","Type":"ContainerDied","Data":"bafc7cd2e69bc49eba568e173fdf9c3499aefad1893df48c60d0ed73a7600ffb"}
Oct 30 23:04:44.633245 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:44.633191    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-2b1e88/simulator-w1-2b1e88-kserve-6c579b99f8-q5wpq"
Oct 30 23:04:44.749363 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:44.749326    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/722f10d8-b7a2-4bf7-a274-9669529ef427-home\" (UniqueName: \"kubernetes.io/empty-dir/722f10d8-b7a2-4bf7-a274-9669529ef427-home\") pod \"722f10d8-b7a2-4bf7-a274-9669529ef427\" (UID: \"722f10d8-b7a2-4bf7-a274-9669529ef427\") "
Oct 30 23:04:44.749542 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:44.749418    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/722f10d8-b7a2-4bf7-a274-9669529ef427-dshm\" (UniqueName: \"kubernetes.io/empty-dir/722f10d8-b7a2-4bf7-a274-9669529ef427-dshm\") pod \"722f10d8-b7a2-4bf7-a274-9669529ef427\" (UID: \"722f10d8-b7a2-4bf7-a274-9669529ef427\") "
Oct 30 23:04:44.749542 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:44.749472    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/722f10d8-b7a2-4bf7-a274-9669529ef427-kube-api-access-p6qdg\" (UniqueName: \"kubernetes.io/projected/722f10d8-b7a2-4bf7-a274-9669529ef427-kube-api-access-p6qdg\") pod \"722f10d8-b7a2-4bf7-a274-9669529ef427\" (UID: \"722f10d8-b7a2-4bf7-a274-9669529ef427\") "
Oct 30 23:04:44.749542 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:44.749504    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/722f10d8-b7a2-4bf7-a274-9669529ef427-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/722f10d8-b7a2-4bf7-a274-9669529ef427-model-cache\") pod \"722f10d8-b7a2-4bf7-a274-9669529ef427\" (UID: \"722f10d8-b7a2-4bf7-a274-9669529ef427\") "
Oct 30 23:04:44.749718 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:44.749577    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/722f10d8-b7a2-4bf7-a274-9669529ef427-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/722f10d8-b7a2-4bf7-a274-9669529ef427-tmp-dir\") pod \"722f10d8-b7a2-4bf7-a274-9669529ef427\" (UID: \"722f10d8-b7a2-4bf7-a274-9669529ef427\") "
Oct 30 23:04:44.749718 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:44.749605    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/722f10d8-b7a2-4bf7-a274-9669529ef427-tls-certs\" (UniqueName: \"kubernetes.io/secret/722f10d8-b7a2-4bf7-a274-9669529ef427-tls-certs\") pod \"722f10d8-b7a2-4bf7-a274-9669529ef427\" (UID: \"722f10d8-b7a2-4bf7-a274-9669529ef427\") "
Oct 30 23:04:44.749718 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:44.749636    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/722f10d8-b7a2-4bf7-a274-9669529ef427-home" pod "722f10d8-b7a2-4bf7-a274-9669529ef427" (UID: "722f10d8-b7a2-4bf7-a274-9669529ef427"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:04:44.749889 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:44.749860    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/722f10d8-b7a2-4bf7-a274-9669529ef427-tmp-dir" pod "722f10d8-b7a2-4bf7-a274-9669529ef427" (UID: "722f10d8-b7a2-4bf7-a274-9669529ef427"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:04:44.749947 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:44.749883    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/722f10d8-b7a2-4bf7-a274-9669529ef427-model-cache" pod "722f10d8-b7a2-4bf7-a274-9669529ef427" (UID: "722f10d8-b7a2-4bf7-a274-9669529ef427"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:04:44.750024 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:44.750008    4477 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/722f10d8-b7a2-4bf7-a274-9669529ef427-tmp-dir\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:04:44.750129 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:44.750029    4477 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/722f10d8-b7a2-4bf7-a274-9669529ef427-home\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:04:44.750129 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:44.750076    4477 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/722f10d8-b7a2-4bf7-a274-9669529ef427-model-cache\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:04:44.751724 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:44.751696    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/722f10d8-b7a2-4bf7-a274-9669529ef427-dshm" pod "722f10d8-b7a2-4bf7-a274-9669529ef427" (UID: "722f10d8-b7a2-4bf7-a274-9669529ef427"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:04:44.751825 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:44.751746    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/722f10d8-b7a2-4bf7-a274-9669529ef427-kube-api-access-p6qdg" pod "722f10d8-b7a2-4bf7-a274-9669529ef427" (UID: "722f10d8-b7a2-4bf7-a274-9669529ef427"). InnerVolumeSpecName "kube-api-access-p6qdg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:04:44.752119 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:44.752099    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/722f10d8-b7a2-4bf7-a274-9669529ef427-tls-certs" pod "722f10d8-b7a2-4bf7-a274-9669529ef427" (UID: "722f10d8-b7a2-4bf7-a274-9669529ef427"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:04:44.851606 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:44.851579    4477 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/722f10d8-b7a2-4bf7-a274-9669529ef427-dshm\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:04:44.851606 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:44.851608    4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6qdg\" (UniqueName: \"kubernetes.io/projected/722f10d8-b7a2-4bf7-a274-9669529ef427-kube-api-access-p6qdg\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:04:44.851785 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:44.851618    4477 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/722f10d8-b7a2-4bf7-a274-9669529ef427-tls-certs\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:04:45.583610 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:45.583578    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-2b1e88/simulator-w1-2b1e88-kserve-6c579b99f8-q5wpq"
Oct 30 23:04:45.584029 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:45.583626    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-2b1e88/simulator-w1-2b1e88-kserve-6c579b99f8-q5wpq" event={"ID":"722f10d8-b7a2-4bf7-a274-9669529ef427","Type":"ContainerDied","Data":"946df0cdeb6afa4a96955a531dcb5a150317269b7cffff39f0b83e1849f759d1"}
Oct 30 23:04:45.584029 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:45.583677    4477 scope.go:122] "RemoveContainer" containerID="bafc7cd2e69bc49eba568e173fdf9c3499aefad1893df48c60d0ed73a7600ffb"
Oct 30 23:04:45.629080 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:45.629052    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-2b1e88/simulator-w1-2b1e88-kserve-6c579b99f8-q5wpq"]
Oct 30 23:04:45.639547 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:45.639526    4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-2b1e88/simulator-w1-2b1e88-kserve-6c579b99f8-q5wpq"]
Oct 30 23:04:45.880960 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:45.880922    4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-09d0d5/distinct-2-w0-09d0d5-kserve-7dcf89b98b-7cjph" podUID="8cbd04b4-bb3e-4bbb-a41a-46131c9de051" containerName="main" containerID="cri-o://557d03a6b2a3f5cc3860ee8a21e35b522211143c14a9f15f58380cf9e364a499" gracePeriod=45
Oct 30 23:04:46.158116 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:46.158092    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-09d0d5/distinct-2-w0-09d0d5-kserve-7dcf89b98b-7cjph"
Oct 30 23:04:46.192668 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:46.192642    4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="722f10d8-b7a2-4bf7-a274-9669529ef427" path="/var/lib/kubelet/pods/722f10d8-b7a2-4bf7-a274-9669529ef427/volumes"
Oct 30 23:04:46.269221 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:46.269186    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8cbd04b4-bb3e-4bbb-a41a-46131c9de051-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8cbd04b4-bb3e-4bbb-a41a-46131c9de051-tmp-dir\") pod \"8cbd04b4-bb3e-4bbb-a41a-46131c9de051\" (UID: \"8cbd04b4-bb3e-4bbb-a41a-46131c9de051\") "
Oct 30 23:04:46.269415 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:46.269256    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8cbd04b4-bb3e-4bbb-a41a-46131c9de051-home\" (UniqueName: \"kubernetes.io/empty-dir/8cbd04b4-bb3e-4bbb-a41a-46131c9de051-home\") pod \"8cbd04b4-bb3e-4bbb-a41a-46131c9de051\" (UID: \"8cbd04b4-bb3e-4bbb-a41a-46131c9de051\") "
Oct 30 23:04:46.269415 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:46.269311    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8cbd04b4-bb3e-4bbb-a41a-46131c9de051-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8cbd04b4-bb3e-4bbb-a41a-46131c9de051-model-cache\") pod \"8cbd04b4-bb3e-4bbb-a41a-46131c9de051\" (UID: \"8cbd04b4-bb3e-4bbb-a41a-46131c9de051\") "
Oct 30 23:04:46.269415 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:46.269340    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cbd04b4-bb3e-4bbb-a41a-46131c9de051-kube-api-access-h5zq8\" (UniqueName: \"kubernetes.io/projected/8cbd04b4-bb3e-4bbb-a41a-46131c9de051-kube-api-access-h5zq8\") pod \"8cbd04b4-bb3e-4bbb-a41a-46131c9de051\" (UID: \"8cbd04b4-bb3e-4bbb-a41a-46131c9de051\") "
Oct 30 23:04:46.269618 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:46.269425    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8cbd04b4-bb3e-4bbb-a41a-46131c9de051-tls-certs\" (UniqueName: \"kubernetes.io/secret/8cbd04b4-bb3e-4bbb-a41a-46131c9de051-tls-certs\") pod \"8cbd04b4-bb3e-4bbb-a41a-46131c9de051\" (UID: \"8cbd04b4-bb3e-4bbb-a41a-46131c9de051\") "
Oct 30 23:04:46.269618 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:46.269473    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8cbd04b4-bb3e-4bbb-a41a-46131c9de051-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8cbd04b4-bb3e-4bbb-a41a-46131c9de051-dshm\") pod \"8cbd04b4-bb3e-4bbb-a41a-46131c9de051\" (UID: \"8cbd04b4-bb3e-4bbb-a41a-46131c9de051\") "
Oct 30 23:04:46.269618 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:46.269556    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cbd04b4-bb3e-4bbb-a41a-46131c9de051-tmp-dir" pod "8cbd04b4-bb3e-4bbb-a41a-46131c9de051" (UID: "8cbd04b4-bb3e-4bbb-a41a-46131c9de051"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:04:46.269782 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:46.269766    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cbd04b4-bb3e-4bbb-a41a-46131c9de051-model-cache" pod "8cbd04b4-bb3e-4bbb-a41a-46131c9de051" (UID: "8cbd04b4-bb3e-4bbb-a41a-46131c9de051"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:04:46.269973 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:46.269948    4477 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8cbd04b4-bb3e-4bbb-a41a-46131c9de051-tmp-dir\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:04:46.270167 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:46.269975    4477 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8cbd04b4-bb3e-4bbb-a41a-46131c9de051-model-cache\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:04:46.270283 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:46.270181    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cbd04b4-bb3e-4bbb-a41a-46131c9de051-home" pod "8cbd04b4-bb3e-4bbb-a41a-46131c9de051" (UID: "8cbd04b4-bb3e-4bbb-a41a-46131c9de051"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:04:46.272337 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:46.272306    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cbd04b4-bb3e-4bbb-a41a-46131c9de051-dshm" pod "8cbd04b4-bb3e-4bbb-a41a-46131c9de051" (UID: "8cbd04b4-bb3e-4bbb-a41a-46131c9de051"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:04:46.272428 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:46.272346    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cbd04b4-bb3e-4bbb-a41a-46131c9de051-kube-api-access-h5zq8" pod "8cbd04b4-bb3e-4bbb-a41a-46131c9de051" (UID: "8cbd04b4-bb3e-4bbb-a41a-46131c9de051"). InnerVolumeSpecName "kube-api-access-h5zq8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:04:46.272428 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:46.272378    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cbd04b4-bb3e-4bbb-a41a-46131c9de051-tls-certs" pod "8cbd04b4-bb3e-4bbb-a41a-46131c9de051" (UID: "8cbd04b4-bb3e-4bbb-a41a-46131c9de051"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:04:46.371367 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:46.371336    4477 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8cbd04b4-bb3e-4bbb-a41a-46131c9de051-tls-certs\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:04:46.371367 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:46.371362    4477 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8cbd04b4-bb3e-4bbb-a41a-46131c9de051-dshm\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:04:46.371367 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:46.371371    4477 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8cbd04b4-bb3e-4bbb-a41a-46131c9de051-home\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:04:46.371588 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:46.371379    4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5zq8\" (UniqueName: \"kubernetes.io/projected/8cbd04b4-bb3e-4bbb-a41a-46131c9de051-kube-api-access-h5zq8\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:04:46.591026 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:46.590927    4477 generic.go:356] "Generic (PLEG): container finished" podID="8cbd04b4-bb3e-4bbb-a41a-46131c9de051" containerID="557d03a6b2a3f5cc3860ee8a21e35b522211143c14a9f15f58380cf9e364a499" exitCode=0
Oct 30 23:04:46.591026 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:46.590976    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-09d0d5/distinct-2-w0-09d0d5-kserve-7dcf89b98b-7cjph"
Oct 30 23:04:46.591586 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:46.591020    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-09d0d5/distinct-2-w0-09d0d5-kserve-7dcf89b98b-7cjph" event={"ID":"8cbd04b4-bb3e-4bbb-a41a-46131c9de051","Type":"ContainerDied","Data":"557d03a6b2a3f5cc3860ee8a21e35b522211143c14a9f15f58380cf9e364a499"}
Oct 30 23:04:46.591586 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:46.591104    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-09d0d5/distinct-2-w0-09d0d5-kserve-7dcf89b98b-7cjph" event={"ID":"8cbd04b4-bb3e-4bbb-a41a-46131c9de051","Type":"ContainerDied","Data":"22c1fed02d98268648e6f0f1a020908fd42d87d3137ae37a17d00f5c0d2b0d54"}
Oct 30 23:04:46.591586 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:46.591129    4477 scope.go:122] "RemoveContainer" containerID="557d03a6b2a3f5cc3860ee8a21e35b522211143c14a9f15f58380cf9e364a499"
Oct 30 23:04:46.600549 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:46.600533    4477 scope.go:122] "RemoveContainer" containerID="557d03a6b2a3f5cc3860ee8a21e35b522211143c14a9f15f58380cf9e364a499"
Oct 30 23:04:46.600810 ip-10-0-130-77 kubenswrapper[4477]: E1030 23:04:46.600788    4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"557d03a6b2a3f5cc3860ee8a21e35b522211143c14a9f15f58380cf9e364a499\": container with ID starting with 557d03a6b2a3f5cc3860ee8a21e35b522211143c14a9f15f58380cf9e364a499 not found: ID does not exist" containerID="557d03a6b2a3f5cc3860ee8a21e35b522211143c14a9f15f58380cf9e364a499"
Oct 30 23:04:46.600860 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:46.600818    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"557d03a6b2a3f5cc3860ee8a21e35b522211143c14a9f15f58380cf9e364a499"} err="failed to get container status \"557d03a6b2a3f5cc3860ee8a21e35b522211143c14a9f15f58380cf9e364a499\": rpc error: code = NotFound desc = could not find container \"557d03a6b2a3f5cc3860ee8a21e35b522211143c14a9f15f58380cf9e364a499\": container with ID starting with 557d03a6b2a3f5cc3860ee8a21e35b522211143c14a9f15f58380cf9e364a499 not found: ID does not exist"
Oct 30 23:04:46.637393 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:46.637372    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-09d0d5/distinct-2-w0-09d0d5-kserve-7dcf89b98b-7cjph"]
Oct 30 23:04:46.647283 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:46.647265    4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-09d0d5/distinct-2-w0-09d0d5-kserve-7dcf89b98b-7cjph"]
Oct 30 23:04:48.193421 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:48.193385    4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8cbd04b4-bb3e-4bbb-a41a-46131c9de051" path="/var/lib/kubelet/pods/8cbd04b4-bb3e-4bbb-a41a-46131c9de051/volumes"
Oct 30 23:04:51.075175 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:51.075145    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6bz92"]
Oct 30 23:04:54.561008 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:54.560969    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cdc122-65f9cc65b8-95mpc"]
Oct 30 23:04:54.561516 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:54.561236    4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cdc122-65f9cc65b8-95mpc" podUID="ec4328f4-72a8-4462-b87c-66415cbc6989" containerName="maas-api" containerID="cri-o://4ebd4aec5ad9ec3ab3f8b862a35aa855b44a8f142c0caeb8959a282d02b1ef82" gracePeriod=30
Oct 30 23:04:54.838274 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:54.838247    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cdc122-65f9cc65b8-95mpc"
Oct 30 23:04:54.964928 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:54.964898    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec4328f4-72a8-4462-b87c-66415cbc6989-kube-api-access-sghrh\" (UniqueName: \"kubernetes.io/projected/ec4328f4-72a8-4462-b87c-66415cbc6989-kube-api-access-sghrh\") pod \"ec4328f4-72a8-4462-b87c-66415cbc6989\" (UID: \"ec4328f4-72a8-4462-b87c-66415cbc6989\") "
Oct 30 23:04:54.965130 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:54.964994    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec4328f4-72a8-4462-b87c-66415cbc6989-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec4328f4-72a8-4462-b87c-66415cbc6989-metrics-certs\") pod \"ec4328f4-72a8-4462-b87c-66415cbc6989\" (UID: \"ec4328f4-72a8-4462-b87c-66415cbc6989\") "
Oct 30 23:04:54.965240 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:54.965123    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec4328f4-72a8-4462-b87c-66415cbc6989-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec4328f4-72a8-4462-b87c-66415cbc6989-openshift-service-ca\") pod \"ec4328f4-72a8-4462-b87c-66415cbc6989\" (UID: \"ec4328f4-72a8-4462-b87c-66415cbc6989\") "
Oct 30 23:04:54.965240 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:54.965160    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec4328f4-72a8-4462-b87c-66415cbc6989-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec4328f4-72a8-4462-b87c-66415cbc6989-maas-api-tls\") pod \"ec4328f4-72a8-4462-b87c-66415cbc6989\" (UID: \"ec4328f4-72a8-4462-b87c-66415cbc6989\") "
Oct 30 23:04:54.965560 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:54.965513    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec4328f4-72a8-4462-b87c-66415cbc6989-openshift-service-ca" pod "ec4328f4-72a8-4462-b87c-66415cbc6989" (UID: "ec4328f4-72a8-4462-b87c-66415cbc6989"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:04:54.967306 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:54.967281    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec4328f4-72a8-4462-b87c-66415cbc6989-metrics-certs" pod "ec4328f4-72a8-4462-b87c-66415cbc6989" (UID: "ec4328f4-72a8-4462-b87c-66415cbc6989"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:04:54.967607 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:54.967580    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec4328f4-72a8-4462-b87c-66415cbc6989-maas-api-tls" pod "ec4328f4-72a8-4462-b87c-66415cbc6989" (UID: "ec4328f4-72a8-4462-b87c-66415cbc6989"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:04:54.967705 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:54.967605    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec4328f4-72a8-4462-b87c-66415cbc6989-kube-api-access-sghrh" pod "ec4328f4-72a8-4462-b87c-66415cbc6989" (UID: "ec4328f4-72a8-4462-b87c-66415cbc6989"). InnerVolumeSpecName "kube-api-access-sghrh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:04:55.078137 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:55.073564    4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sghrh\" (UniqueName: \"kubernetes.io/projected/ec4328f4-72a8-4462-b87c-66415cbc6989-kube-api-access-sghrh\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:04:55.078137 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:55.073601    4477 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec4328f4-72a8-4462-b87c-66415cbc6989-metrics-certs\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:04:55.078137 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:55.073616    4477 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec4328f4-72a8-4462-b87c-66415cbc6989-openshift-service-ca\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:04:55.078137 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:55.073630    4477 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec4328f4-72a8-4462-b87c-66415cbc6989-maas-api-tls\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:04:55.643919 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:55.643882    4477 generic.go:356] "Generic (PLEG): container finished" podID="ec4328f4-72a8-4462-b87c-66415cbc6989" containerID="4ebd4aec5ad9ec3ab3f8b862a35aa855b44a8f142c0caeb8959a282d02b1ef82" exitCode=0
Oct 30 23:04:55.644348 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:55.643934    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cdc122-65f9cc65b8-95mpc"
Oct 30 23:04:55.644348 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:55.643966    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cdc122-65f9cc65b8-95mpc" event={"ID":"ec4328f4-72a8-4462-b87c-66415cbc6989","Type":"ContainerDied","Data":"4ebd4aec5ad9ec3ab3f8b862a35aa855b44a8f142c0caeb8959a282d02b1ef82"}
Oct 30 23:04:55.644348 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:55.644020    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cdc122-65f9cc65b8-95mpc" event={"ID":"ec4328f4-72a8-4462-b87c-66415cbc6989","Type":"ContainerDied","Data":"11081b1468b302a580a3d1fa2f7c6588854dd285672790bf33cf51ae12433d3b"}
Oct 30 23:04:55.644348 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:55.644056    4477 scope.go:122] "RemoveContainer" containerID="4ebd4aec5ad9ec3ab3f8b862a35aa855b44a8f142c0caeb8959a282d02b1ef82"
Oct 30 23:04:55.654824 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:55.654802    4477 scope.go:122] "RemoveContainer" containerID="4ebd4aec5ad9ec3ab3f8b862a35aa855b44a8f142c0caeb8959a282d02b1ef82"
Oct 30 23:04:55.655119 ip-10-0-130-77 kubenswrapper[4477]: E1030 23:04:55.655095    4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ebd4aec5ad9ec3ab3f8b862a35aa855b44a8f142c0caeb8959a282d02b1ef82\": container with ID starting with 4ebd4aec5ad9ec3ab3f8b862a35aa855b44a8f142c0caeb8959a282d02b1ef82 not found: ID does not exist" containerID="4ebd4aec5ad9ec3ab3f8b862a35aa855b44a8f142c0caeb8959a282d02b1ef82"
Oct 30 23:04:55.655184 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:55.655127    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ebd4aec5ad9ec3ab3f8b862a35aa855b44a8f142c0caeb8959a282d02b1ef82"} err="failed to get container status \"4ebd4aec5ad9ec3ab3f8b862a35aa855b44a8f142c0caeb8959a282d02b1ef82\": rpc error: code = NotFound desc = could not find container \"4ebd4aec5ad9ec3ab3f8b862a35aa855b44a8f142c0caeb8959a282d02b1ef82\": container with ID starting with 4ebd4aec5ad9ec3ab3f8b862a35aa855b44a8f142c0caeb8959a282d02b1ef82 not found: ID does not exist"
Oct 30 23:04:55.690758 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:55.690726    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cdc122-65f9cc65b8-95mpc"]
Oct 30 23:04:55.701032 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:55.701011    4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cdc122-65f9cc65b8-95mpc"]
Oct 30 23:04:56.193957 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:04:56.193923    4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec4328f4-72a8-4462-b87c-66415cbc6989" path="/var/lib/kubelet/pods/ec4328f4-72a8-4462-b87c-66415cbc6989/volumes"
Oct 30 23:04:56.266379 ip-10-0-130-77 kubenswrapper[4477]: E1030 23:04:56.266347    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 23:05:03.109189 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:03.109156    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-2b1e88-6b57666d9d-7lhjc"]
Oct 30 23:05:03.109668 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:03.109377    4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-2b1e88-6b57666d9d-7lhjc" podUID="c67bfb3e-cc4c-40a2-9d8f-bc36f48f2a2f" containerName="maas-api" containerID="cri-o://da2e50f1e9a9c0c4566dcdacf20718471f0e990d6d6c8177a08d176c3f42338e" gracePeriod=30
Oct 30 23:05:03.377878 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:03.377848    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-2b1e88-6b57666d9d-7lhjc"
Oct 30 23:05:03.463630 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:03.463594    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c67bfb3e-cc4c-40a2-9d8f-bc36f48f2a2f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c67bfb3e-cc4c-40a2-9d8f-bc36f48f2a2f-maas-api-tls\") pod \"c67bfb3e-cc4c-40a2-9d8f-bc36f48f2a2f\" (UID: \"c67bfb3e-cc4c-40a2-9d8f-bc36f48f2a2f\") "
Oct 30 23:05:03.463794 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:03.463706    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c67bfb3e-cc4c-40a2-9d8f-bc36f48f2a2f-kube-api-access-dmnth\" (UniqueName: \"kubernetes.io/projected/c67bfb3e-cc4c-40a2-9d8f-bc36f48f2a2f-kube-api-access-dmnth\") pod \"c67bfb3e-cc4c-40a2-9d8f-bc36f48f2a2f\" (UID: \"c67bfb3e-cc4c-40a2-9d8f-bc36f48f2a2f\") "
Oct 30 23:05:03.463794 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:03.463753    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c67bfb3e-cc4c-40a2-9d8f-bc36f48f2a2f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c67bfb3e-cc4c-40a2-9d8f-bc36f48f2a2f-openshift-service-ca\") pod \"c67bfb3e-cc4c-40a2-9d8f-bc36f48f2a2f\" (UID: \"c67bfb3e-cc4c-40a2-9d8f-bc36f48f2a2f\") "
Oct 30 23:05:03.463884 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:03.463824    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c67bfb3e-cc4c-40a2-9d8f-bc36f48f2a2f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c67bfb3e-cc4c-40a2-9d8f-bc36f48f2a2f-metrics-certs\") pod \"c67bfb3e-cc4c-40a2-9d8f-bc36f48f2a2f\" (UID: \"c67bfb3e-cc4c-40a2-9d8f-bc36f48f2a2f\") "
Oct 30 23:05:03.464108 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:03.464081    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c67bfb3e-cc4c-40a2-9d8f-bc36f48f2a2f-openshift-service-ca" pod "c67bfb3e-cc4c-40a2-9d8f-bc36f48f2a2f" (UID: "c67bfb3e-cc4c-40a2-9d8f-bc36f48f2a2f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:05:03.464315 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:03.464293    4477 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c67bfb3e-cc4c-40a2-9d8f-bc36f48f2a2f-openshift-service-ca\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:05:03.465817 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:03.465795    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c67bfb3e-cc4c-40a2-9d8f-bc36f48f2a2f-maas-api-tls" pod "c67bfb3e-cc4c-40a2-9d8f-bc36f48f2a2f" (UID: "c67bfb3e-cc4c-40a2-9d8f-bc36f48f2a2f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:05:03.465896 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:03.465877    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c67bfb3e-cc4c-40a2-9d8f-bc36f48f2a2f-kube-api-access-dmnth" pod "c67bfb3e-cc4c-40a2-9d8f-bc36f48f2a2f" (UID: "c67bfb3e-cc4c-40a2-9d8f-bc36f48f2a2f"). InnerVolumeSpecName "kube-api-access-dmnth". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:05:03.465941 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:03.465892    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c67bfb3e-cc4c-40a2-9d8f-bc36f48f2a2f-metrics-certs" pod "c67bfb3e-cc4c-40a2-9d8f-bc36f48f2a2f" (UID: "c67bfb3e-cc4c-40a2-9d8f-bc36f48f2a2f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:05:03.565558 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:03.565517    4477 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c67bfb3e-cc4c-40a2-9d8f-bc36f48f2a2f-metrics-certs\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:05:03.565558 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:03.565555    4477 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c67bfb3e-cc4c-40a2-9d8f-bc36f48f2a2f-maas-api-tls\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:05:03.565753 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:03.565567    4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmnth\" (UniqueName: \"kubernetes.io/projected/c67bfb3e-cc4c-40a2-9d8f-bc36f48f2a2f-kube-api-access-dmnth\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:05:03.686822 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:03.686785    4477 generic.go:356] "Generic (PLEG): container finished" podID="c67bfb3e-cc4c-40a2-9d8f-bc36f48f2a2f" containerID="da2e50f1e9a9c0c4566dcdacf20718471f0e990d6d6c8177a08d176c3f42338e" exitCode=0
Oct 30 23:05:03.687022 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:03.686839    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-2b1e88-6b57666d9d-7lhjc"
Oct 30 23:05:03.687022 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:03.686865    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-2b1e88-6b57666d9d-7lhjc" event={"ID":"c67bfb3e-cc4c-40a2-9d8f-bc36f48f2a2f","Type":"ContainerDied","Data":"da2e50f1e9a9c0c4566dcdacf20718471f0e990d6d6c8177a08d176c3f42338e"}
Oct 30 23:05:03.687022 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:03.686920    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-2b1e88-6b57666d9d-7lhjc" event={"ID":"c67bfb3e-cc4c-40a2-9d8f-bc36f48f2a2f","Type":"ContainerDied","Data":"c5ea3c2fbdfea8a022d8becd6152dc48c8a73d7c332ae21cbd854671be2e2be2"}
Oct 30 23:05:03.687022 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:03.686942    4477 scope.go:122] "RemoveContainer" containerID="da2e50f1e9a9c0c4566dcdacf20718471f0e990d6d6c8177a08d176c3f42338e"
Oct 30 23:05:03.696946 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:03.696928    4477 scope.go:122] "RemoveContainer" containerID="da2e50f1e9a9c0c4566dcdacf20718471f0e990d6d6c8177a08d176c3f42338e"
Oct 30 23:05:03.697257 ip-10-0-130-77 kubenswrapper[4477]: E1030 23:05:03.697236    4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da2e50f1e9a9c0c4566dcdacf20718471f0e990d6d6c8177a08d176c3f42338e\": container with ID starting with da2e50f1e9a9c0c4566dcdacf20718471f0e990d6d6c8177a08d176c3f42338e not found: ID does not exist" containerID="da2e50f1e9a9c0c4566dcdacf20718471f0e990d6d6c8177a08d176c3f42338e"
Oct 30 23:05:03.697311 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:03.697268    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da2e50f1e9a9c0c4566dcdacf20718471f0e990d6d6c8177a08d176c3f42338e"} err="failed to get container status \"da2e50f1e9a9c0c4566dcdacf20718471f0e990d6d6c8177a08d176c3f42338e\": rpc error: code = NotFound desc = could not find container \"da2e50f1e9a9c0c4566dcdacf20718471f0e990d6d6c8177a08d176c3f42338e\": container with ID starting with da2e50f1e9a9c0c4566dcdacf20718471f0e990d6d6c8177a08d176c3f42338e not found: ID does not exist"
Oct 30 23:05:03.736815 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:03.736793    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-2b1e88-6b57666d9d-7lhjc"]
Oct 30 23:05:03.743403 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:03.743381    4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-2b1e88-6b57666d9d-7lhjc"]
Oct 30 23:05:04.192289 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:04.192259    4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c67bfb3e-cc4c-40a2-9d8f-bc36f48f2a2f" path="/var/lib/kubelet/pods/c67bfb3e-cc4c-40a2-9d8f-bc36f48f2a2f/volumes"
Oct 30 23:05:04.761463 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:04.761427    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-6bvgh"]
Oct 30 23:05:04.761736 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:04.761683    4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-6bvgh" podUID="18a561b5-289d-4401-8377-0be0130d658d" containerName="istio-proxy" containerID="cri-o://4e1fa27a8e840f0dd8b7546ecdbfb0351b1b94e20a4a1801b43603755d4ee49d" gracePeriod=30
Oct 30 23:05:05.879428 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:05.879363    4477 patch_prober.go:29] interesting pod/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-6bvgh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 23:05:05.879932 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:05.879479    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-6bvgh" podUID="18a561b5-289d-4401-8377-0be0130d658d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 23:05:10.049447 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:10.049418    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-6bvgh"
Oct 30 23:05:10.137681 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:10.137647    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18a561b5-289d-4401-8377-0be0130d658d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18a561b5-289d-4401-8377-0be0130d658d-workload-certs\") pod \"18a561b5-289d-4401-8377-0be0130d658d\" (UID: \"18a561b5-289d-4401-8377-0be0130d658d\") "
Oct 30 23:05:10.137849 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:10.137710    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18a561b5-289d-4401-8377-0be0130d658d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18a561b5-289d-4401-8377-0be0130d658d-istio-data\") pod \"18a561b5-289d-4401-8377-0be0130d658d\" (UID: \"18a561b5-289d-4401-8377-0be0130d658d\") "
Oct 30 23:05:10.137849 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:10.137839    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18a561b5-289d-4401-8377-0be0130d658d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18a561b5-289d-4401-8377-0be0130d658d-istio-envoy\") pod \"18a561b5-289d-4401-8377-0be0130d658d\" (UID: \"18a561b5-289d-4401-8377-0be0130d658d\") "
Oct 30 23:05:10.137974 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:10.137879    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18a561b5-289d-4401-8377-0be0130d658d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18a561b5-289d-4401-8377-0be0130d658d-istiod-ca-cert\") pod \"18a561b5-289d-4401-8377-0be0130d658d\" (UID: \"18a561b5-289d-4401-8377-0be0130d658d\") "
Oct 30 23:05:10.137974 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:10.137932    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18a561b5-289d-4401-8377-0be0130d658d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18a561b5-289d-4401-8377-0be0130d658d-credential-socket\") pod \"18a561b5-289d-4401-8377-0be0130d658d\" (UID: \"18a561b5-289d-4401-8377-0be0130d658d\") "
Oct 30 23:05:10.138124 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:10.137973    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18a561b5-289d-4401-8377-0be0130d658d-workload-certs" pod "18a561b5-289d-4401-8377-0be0130d658d" (UID: "18a561b5-289d-4401-8377-0be0130d658d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:05:10.138124 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:10.137986    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18a561b5-289d-4401-8377-0be0130d658d-istio-data" pod "18a561b5-289d-4401-8377-0be0130d658d" (UID: "18a561b5-289d-4401-8377-0be0130d658d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:05:10.138124 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:10.137982    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18a561b5-289d-4401-8377-0be0130d658d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18a561b5-289d-4401-8377-0be0130d658d-workload-socket\") pod \"18a561b5-289d-4401-8377-0be0130d658d\" (UID: \"18a561b5-289d-4401-8377-0be0130d658d\") "
Oct 30 23:05:10.138124 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:10.138076    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18a561b5-289d-4401-8377-0be0130d658d-kube-api-access-vxbhf\" (UniqueName: \"kubernetes.io/projected/18a561b5-289d-4401-8377-0be0130d658d-kube-api-access-vxbhf\") pod \"18a561b5-289d-4401-8377-0be0130d658d\" (UID: \"18a561b5-289d-4401-8377-0be0130d658d\") "
Oct 30 23:05:10.138331 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:10.138136    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/18a561b5-289d-4401-8377-0be0130d658d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18a561b5-289d-4401-8377-0be0130d658d-istio-podinfo\") pod \"18a561b5-289d-4401-8377-0be0130d658d\" (UID: \"18a561b5-289d-4401-8377-0be0130d658d\") "
Oct 30 23:05:10.138331 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:10.138173    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18a561b5-289d-4401-8377-0be0130d658d-istio-token\" (UniqueName: \"kubernetes.io/projected/18a561b5-289d-4401-8377-0be0130d658d-istio-token\") pod \"18a561b5-289d-4401-8377-0be0130d658d\" (UID: \"18a561b5-289d-4401-8377-0be0130d658d\") "
Oct 30 23:05:10.138449 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:10.138403    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18a561b5-289d-4401-8377-0be0130d658d-credential-socket" pod "18a561b5-289d-4401-8377-0be0130d658d" (UID: "18a561b5-289d-4401-8377-0be0130d658d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:05:10.138449 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:10.138432    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18a561b5-289d-4401-8377-0be0130d658d-istiod-ca-cert" pod "18a561b5-289d-4401-8377-0be0130d658d" (UID: "18a561b5-289d-4401-8377-0be0130d658d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:05:10.138613 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:10.138596    4477 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18a561b5-289d-4401-8377-0be0130d658d-credential-socket\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:05:10.138680 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:10.138614    4477 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18a561b5-289d-4401-8377-0be0130d658d-workload-certs\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:05:10.138680 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:10.138623    4477 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18a561b5-289d-4401-8377-0be0130d658d-istio-data\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:05:10.138680 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:10.138631    4477 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18a561b5-289d-4401-8377-0be0130d658d-istiod-ca-cert\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:05:10.139097 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:10.139029    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18a561b5-289d-4401-8377-0be0130d658d-workload-socket" pod "18a561b5-289d-4401-8377-0be0130d658d" (UID: "18a561b5-289d-4401-8377-0be0130d658d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:05:10.140271 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:10.140247    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18a561b5-289d-4401-8377-0be0130d658d-istio-envoy" pod "18a561b5-289d-4401-8377-0be0130d658d" (UID: "18a561b5-289d-4401-8377-0be0130d658d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:05:10.140685 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:10.140667    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/18a561b5-289d-4401-8377-0be0130d658d-istio-podinfo" pod "18a561b5-289d-4401-8377-0be0130d658d" (UID: "18a561b5-289d-4401-8377-0be0130d658d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:05:10.140867 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:10.140842    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18a561b5-289d-4401-8377-0be0130d658d-istio-token" pod "18a561b5-289d-4401-8377-0be0130d658d" (UID: "18a561b5-289d-4401-8377-0be0130d658d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:05:10.140952 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:10.140905    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18a561b5-289d-4401-8377-0be0130d658d-kube-api-access-vxbhf" pod "18a561b5-289d-4401-8377-0be0130d658d" (UID: "18a561b5-289d-4401-8377-0be0130d658d"). InnerVolumeSpecName "kube-api-access-vxbhf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:05:10.239856 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:10.239835    4477 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18a561b5-289d-4401-8377-0be0130d658d-istio-envoy\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:05:10.239976 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:10.239864    4477 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18a561b5-289d-4401-8377-0be0130d658d-workload-socket\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:05:10.239976 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:10.239882    4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxbhf\" (UniqueName: \"kubernetes.io/projected/18a561b5-289d-4401-8377-0be0130d658d-kube-api-access-vxbhf\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:05:10.239976 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:10.239892    4477 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18a561b5-289d-4401-8377-0be0130d658d-istio-podinfo\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:05:10.239976 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:10.239901    4477 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18a561b5-289d-4401-8377-0be0130d658d-istio-token\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:05:10.622370 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:10.622332    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6bz92"]
Oct 30 23:05:10.739003 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:10.738964    4477 generic.go:356] "Generic (PLEG): container finished" podID="18a561b5-289d-4401-8377-0be0130d658d" containerID="4e1fa27a8e840f0dd8b7546ecdbfb0351b1b94e20a4a1801b43603755d4ee49d" exitCode=0
Oct 30 23:05:10.739229 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:10.739029    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-6bvgh" event={"ID":"18a561b5-289d-4401-8377-0be0130d658d","Type":"ContainerDied","Data":"4e1fa27a8e840f0dd8b7546ecdbfb0351b1b94e20a4a1801b43603755d4ee49d"}
Oct 30 23:05:10.739229 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:10.739074    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-6bvgh"
Oct 30 23:05:10.739229 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:10.739100    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-6bvgh" event={"ID":"18a561b5-289d-4401-8377-0be0130d658d","Type":"ContainerDied","Data":"4e916028dbaafb148f091bedc1b985d89be93d6e51025350f144ff53e34d83d0"}
Oct 30 23:05:10.739229 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:10.739131    4477 scope.go:122] "RemoveContainer" containerID="4e1fa27a8e840f0dd8b7546ecdbfb0351b1b94e20a4a1801b43603755d4ee49d"
Oct 30 23:05:10.753627 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:10.753606    4477 scope.go:122] "RemoveContainer" containerID="4e1fa27a8e840f0dd8b7546ecdbfb0351b1b94e20a4a1801b43603755d4ee49d"
Oct 30 23:05:10.753954 ip-10-0-130-77 kubenswrapper[4477]: E1030 23:05:10.753932    4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e1fa27a8e840f0dd8b7546ecdbfb0351b1b94e20a4a1801b43603755d4ee49d\": container with ID starting with 4e1fa27a8e840f0dd8b7546ecdbfb0351b1b94e20a4a1801b43603755d4ee49d not found: ID does not exist" containerID="4e1fa27a8e840f0dd8b7546ecdbfb0351b1b94e20a4a1801b43603755d4ee49d"
Oct 30 23:05:10.754032 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:10.753966    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e1fa27a8e840f0dd8b7546ecdbfb0351b1b94e20a4a1801b43603755d4ee49d"} err="failed to get container status \"4e1fa27a8e840f0dd8b7546ecdbfb0351b1b94e20a4a1801b43603755d4ee49d\": rpc error: code = NotFound desc = could not find container \"4e1fa27a8e840f0dd8b7546ecdbfb0351b1b94e20a4a1801b43603755d4ee49d\": container with ID starting with 4e1fa27a8e840f0dd8b7546ecdbfb0351b1b94e20a4a1801b43603755d4ee49d not found: ID does not exist"
Oct 30 23:05:10.804066 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:10.803563    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-6bvgh"]
Oct 30 23:05:10.816861 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:10.816822    4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-cdc122-openshift-default-7b7d59b4dc-6bvgh"]
Oct 30 23:05:12.199830 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:12.199791    4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18a561b5-289d-4401-8377-0be0130d658d" path="/var/lib/kubelet/pods/18a561b5-289d-4401-8377-0be0130d658d/volumes"
Oct 30 23:05:13.703862 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:13.703825    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-cjkc9"]
Oct 30 23:05:13.704812 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:13.704771    4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-cjkc9" podUID="e8a3aba3-7d06-4fd5-8106-2d02250f1da2" containerName="istio-proxy" containerID="cri-o://019a986e536c9e2adbd8f4c7a9976538e38f3215dbb98967f7b5f12eb6167d2a" gracePeriod=30
Oct 30 23:05:14.052590 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:14.052511    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-pmswk"]
Oct 30 23:05:14.052794 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:14.052765    4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-pmswk" podUID="9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7" containerName="istio-proxy" containerID="cri-o://ed35cf609bf857a95d6417e89e7792109358a7da9992f72952de20867bf7ce4c" gracePeriod=30
Oct 30 23:05:14.807566 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:14.807530    4477 patch_prober.go:29] interesting pod/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-pmswk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 23:05:14.808029 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:14.807585    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-pmswk" podUID="9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 23:05:15.451654 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:15.451614    4477 patch_prober.go:29] interesting pod/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-cjkc9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 23:05:15.451823 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:15.451663    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-cjkc9" podUID="e8a3aba3-7d06-4fd5-8106-2d02250f1da2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 23:05:18.961923 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:18.961898    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-cjkc9"
Oct 30 23:05:19.039905 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:19.039842    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8a3aba3-7d06-4fd5-8106-2d02250f1da2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8a3aba3-7d06-4fd5-8106-2d02250f1da2-istio-data\") pod \"e8a3aba3-7d06-4fd5-8106-2d02250f1da2\" (UID: \"e8a3aba3-7d06-4fd5-8106-2d02250f1da2\") "
Oct 30 23:05:19.039905 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:19.039877    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8a3aba3-7d06-4fd5-8106-2d02250f1da2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8a3aba3-7d06-4fd5-8106-2d02250f1da2-istio-envoy\") pod \"e8a3aba3-7d06-4fd5-8106-2d02250f1da2\" (UID: \"e8a3aba3-7d06-4fd5-8106-2d02250f1da2\") "
Oct 30 23:05:19.040122 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:19.039952    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e8a3aba3-7d06-4fd5-8106-2d02250f1da2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8a3aba3-7d06-4fd5-8106-2d02250f1da2-istio-podinfo\") pod \"e8a3aba3-7d06-4fd5-8106-2d02250f1da2\" (UID: \"e8a3aba3-7d06-4fd5-8106-2d02250f1da2\") "
Oct 30 23:05:19.040122 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:19.039979    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8a3aba3-7d06-4fd5-8106-2d02250f1da2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8a3aba3-7d06-4fd5-8106-2d02250f1da2-workload-socket\") pod \"e8a3aba3-7d06-4fd5-8106-2d02250f1da2\" (UID: \"e8a3aba3-7d06-4fd5-8106-2d02250f1da2\") "
Oct 30 23:05:19.040122 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:19.040012    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8a3aba3-7d06-4fd5-8106-2d02250f1da2-kube-api-access-dqr4w\" (UniqueName: \"kubernetes.io/projected/e8a3aba3-7d06-4fd5-8106-2d02250f1da2-kube-api-access-dqr4w\") pod \"e8a3aba3-7d06-4fd5-8106-2d02250f1da2\" (UID: \"e8a3aba3-7d06-4fd5-8106-2d02250f1da2\") "
Oct 30 23:05:19.040122 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:19.040058    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8a3aba3-7d06-4fd5-8106-2d02250f1da2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8a3aba3-7d06-4fd5-8106-2d02250f1da2-workload-certs\") pod \"e8a3aba3-7d06-4fd5-8106-2d02250f1da2\" (UID: \"e8a3aba3-7d06-4fd5-8106-2d02250f1da2\") "
Oct 30 23:05:19.040343 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:19.040134    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8a3aba3-7d06-4fd5-8106-2d02250f1da2-istio-token\" (UniqueName: \"kubernetes.io/projected/e8a3aba3-7d06-4fd5-8106-2d02250f1da2-istio-token\") pod \"e8a3aba3-7d06-4fd5-8106-2d02250f1da2\" (UID: \"e8a3aba3-7d06-4fd5-8106-2d02250f1da2\") "
Oct 30 23:05:19.040343 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:19.040178    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8a3aba3-7d06-4fd5-8106-2d02250f1da2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8a3aba3-7d06-4fd5-8106-2d02250f1da2-istiod-ca-cert\") pod \"e8a3aba3-7d06-4fd5-8106-2d02250f1da2\" (UID: \"e8a3aba3-7d06-4fd5-8106-2d02250f1da2\") "
Oct 30 23:05:19.040343 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:19.040180    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8a3aba3-7d06-4fd5-8106-2d02250f1da2-istio-data" pod "e8a3aba3-7d06-4fd5-8106-2d02250f1da2" (UID: "e8a3aba3-7d06-4fd5-8106-2d02250f1da2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:05:19.040343 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:19.040279    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8a3aba3-7d06-4fd5-8106-2d02250f1da2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8a3aba3-7d06-4fd5-8106-2d02250f1da2-credential-socket\") pod \"e8a3aba3-7d06-4fd5-8106-2d02250f1da2\" (UID: \"e8a3aba3-7d06-4fd5-8106-2d02250f1da2\") "
Oct 30 23:05:19.040566 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:19.040510    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8a3aba3-7d06-4fd5-8106-2d02250f1da2-istiod-ca-cert" pod "e8a3aba3-7d06-4fd5-8106-2d02250f1da2" (UID: "e8a3aba3-7d06-4fd5-8106-2d02250f1da2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:05:19.040798 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:19.040770    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8a3aba3-7d06-4fd5-8106-2d02250f1da2-credential-socket" pod "e8a3aba3-7d06-4fd5-8106-2d02250f1da2" (UID: "e8a3aba3-7d06-4fd5-8106-2d02250f1da2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:05:19.040899 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:19.040773    4477 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8a3aba3-7d06-4fd5-8106-2d02250f1da2-istio-data\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:05:19.040899 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:19.040791    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8a3aba3-7d06-4fd5-8106-2d02250f1da2-workload-certs" pod "e8a3aba3-7d06-4fd5-8106-2d02250f1da2" (UID: "e8a3aba3-7d06-4fd5-8106-2d02250f1da2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:05:19.040899 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:19.040821    4477 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8a3aba3-7d06-4fd5-8106-2d02250f1da2-istiod-ca-cert\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:05:19.040899 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:19.040871    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8a3aba3-7d06-4fd5-8106-2d02250f1da2-workload-socket" pod "e8a3aba3-7d06-4fd5-8106-2d02250f1da2" (UID: "e8a3aba3-7d06-4fd5-8106-2d02250f1da2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:05:19.042655 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:19.042622    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8a3aba3-7d06-4fd5-8106-2d02250f1da2-istio-envoy" pod "e8a3aba3-7d06-4fd5-8106-2d02250f1da2" (UID: "e8a3aba3-7d06-4fd5-8106-2d02250f1da2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:05:19.042784 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:19.042671    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e8a3aba3-7d06-4fd5-8106-2d02250f1da2-istio-podinfo" pod "e8a3aba3-7d06-4fd5-8106-2d02250f1da2" (UID: "e8a3aba3-7d06-4fd5-8106-2d02250f1da2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:05:19.043024 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:19.042991    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8a3aba3-7d06-4fd5-8106-2d02250f1da2-istio-token" pod "e8a3aba3-7d06-4fd5-8106-2d02250f1da2" (UID: "e8a3aba3-7d06-4fd5-8106-2d02250f1da2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:05:19.043157 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:19.043031    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8a3aba3-7d06-4fd5-8106-2d02250f1da2-kube-api-access-dqr4w" pod "e8a3aba3-7d06-4fd5-8106-2d02250f1da2" (UID: "e8a3aba3-7d06-4fd5-8106-2d02250f1da2"). InnerVolumeSpecName "kube-api-access-dqr4w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:05:19.142323 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:19.142295    4477 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8a3aba3-7d06-4fd5-8106-2d02250f1da2-workload-certs\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:05:19.142323 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:19.142320    4477 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8a3aba3-7d06-4fd5-8106-2d02250f1da2-istio-token\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:05:19.142470 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:19.142332    4477 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8a3aba3-7d06-4fd5-8106-2d02250f1da2-credential-socket\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:05:19.142470 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:19.142346    4477 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8a3aba3-7d06-4fd5-8106-2d02250f1da2-istio-envoy\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:05:19.142470 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:19.142358    4477 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8a3aba3-7d06-4fd5-8106-2d02250f1da2-istio-podinfo\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:05:19.142470 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:19.142370    4477 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8a3aba3-7d06-4fd5-8106-2d02250f1da2-workload-socket\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:05:19.142470 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:19.142382    4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqr4w\" (UniqueName: \"kubernetes.io/projected/e8a3aba3-7d06-4fd5-8106-2d02250f1da2-kube-api-access-dqr4w\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:05:19.291915 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:19.291856    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-pmswk"
Oct 30 23:05:19.344008 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:19.343978    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7-workload-certs\") pod \"9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7\" (UID: \"9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7\") "
Oct 30 23:05:19.344136 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:19.344026    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7-credential-socket\") pod \"9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7\" (UID: \"9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7\") "
Oct 30 23:05:19.344136 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:19.344075    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7-istio-token\" (UniqueName: \"kubernetes.io/projected/9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7-istio-token\") pod \"9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7\" (UID: \"9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7\") "
Oct 30 23:05:19.344136 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:19.344125    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7-istio-envoy\") pod \"9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7\" (UID: \"9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7\") "
Oct 30 23:05:19.344280 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:19.344199    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7-istiod-ca-cert\") pod \"9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7\" (UID: \"9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7\") "
Oct 30 23:05:19.344280 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:19.344254    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7-workload-certs" pod "9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7" (UID: "9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:05:19.344353 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:19.344283    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7-kube-api-access-89skf\" (UniqueName: \"kubernetes.io/projected/9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7-kube-api-access-89skf\") pod \"9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7\" (UID: \"9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7\") "
Oct 30 23:05:19.344393 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:19.344349    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7-credential-socket" pod "9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7" (UID: "9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:05:19.344393 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:19.344377    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7-istio-podinfo\") pod \"9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7\" (UID: \"9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7\") "
Oct 30 23:05:19.344507 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:19.344445    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7-istio-data\") pod \"9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7\" (UID: \"9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7\") "
Oct 30 23:05:19.344507 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:19.344492    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7-workload-socket\") pod \"9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7\" (UID: \"9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7\") "
Oct 30 23:05:19.344793 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:19.344766    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7-istio-data" pod "9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7" (UID: "9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:05:19.344793 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:19.344771    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7-istiod-ca-cert" pod "9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7" (UID: "9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:05:19.344793 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:19.344793    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7-workload-socket" pod "9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7" (UID: "9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:05:19.345081 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:19.344843    4477 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7-workload-certs\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:05:19.345081 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:19.344864    4477 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7-credential-socket\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:05:19.346439 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:19.346416    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7-kube-api-access-89skf" pod "9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7" (UID: "9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7"). InnerVolumeSpecName "kube-api-access-89skf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:05:19.346622 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:19.346598    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7-istio-podinfo" pod "9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7" (UID: "9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:05:19.346874 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:19.346855    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7-istio-envoy" pod "9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7" (UID: "9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:05:19.346934 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:19.346897    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7-istio-token" pod "9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7" (UID: "9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:05:19.378231 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:19.378204    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e14e00-6bfcb896b7-xq4xg"]
Oct 30 23:05:19.378435 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:19.378400    4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e14e00-6bfcb896b7-xq4xg" podUID="8157c822-ab10-4925-b3c9-6211dff26119" containerName="maas-api" containerID="cri-o://c715d01dee2aaf025072e7a92e611d31396fa32e4b4a9a4b55193f32222254e0" gracePeriod=30
Oct 30 23:05:19.446186 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:19.446147    4477 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7-istio-envoy\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:05:19.446186 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:19.446185    4477 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7-istiod-ca-cert\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:05:19.446186 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:19.446200    4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89skf\" (UniqueName: \"kubernetes.io/projected/9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7-kube-api-access-89skf\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:05:19.446817 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:19.446214    4477 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7-istio-podinfo\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:05:19.446817 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:19.446227    4477 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7-istio-data\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:05:19.446817 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:19.446239    4477 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7-workload-socket\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:05:19.446817 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:19.446251    4477 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7-istio-token\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:05:19.624839 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:19.624812    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e14e00-6bfcb896b7-xq4xg"
Oct 30 23:05:19.750010 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:19.749982    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8157c822-ab10-4925-b3c9-6211dff26119-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8157c822-ab10-4925-b3c9-6211dff26119-openshift-service-ca\") pod \"8157c822-ab10-4925-b3c9-6211dff26119\" (UID: \"8157c822-ab10-4925-b3c9-6211dff26119\") "
Oct 30 23:05:19.750155 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:19.750033    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8157c822-ab10-4925-b3c9-6211dff26119-kube-api-access-fhvkt\" (UniqueName: \"kubernetes.io/projected/8157c822-ab10-4925-b3c9-6211dff26119-kube-api-access-fhvkt\") pod \"8157c822-ab10-4925-b3c9-6211dff26119\" (UID: \"8157c822-ab10-4925-b3c9-6211dff26119\") "
Oct 30 23:05:19.750155 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:19.750127    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8157c822-ab10-4925-b3c9-6211dff26119-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8157c822-ab10-4925-b3c9-6211dff26119-maas-api-tls\") pod \"8157c822-ab10-4925-b3c9-6211dff26119\" (UID: \"8157c822-ab10-4925-b3c9-6211dff26119\") "
Oct 30 23:05:19.750281 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:19.750172    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8157c822-ab10-4925-b3c9-6211dff26119-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8157c822-ab10-4925-b3c9-6211dff26119-metrics-certs\") pod \"8157c822-ab10-4925-b3c9-6211dff26119\" (UID: \"8157c822-ab10-4925-b3c9-6211dff26119\") "
Oct 30 23:05:19.750456 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:19.750434    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8157c822-ab10-4925-b3c9-6211dff26119-openshift-service-ca" pod "8157c822-ab10-4925-b3c9-6211dff26119" (UID: "8157c822-ab10-4925-b3c9-6211dff26119"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:05:19.750553 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:19.750538    4477 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8157c822-ab10-4925-b3c9-6211dff26119-openshift-service-ca\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:05:19.752266 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:19.752237    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8157c822-ab10-4925-b3c9-6211dff26119-metrics-certs" pod "8157c822-ab10-4925-b3c9-6211dff26119" (UID: "8157c822-ab10-4925-b3c9-6211dff26119"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:05:19.752381 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:19.752313    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8157c822-ab10-4925-b3c9-6211dff26119-kube-api-access-fhvkt" pod "8157c822-ab10-4925-b3c9-6211dff26119" (UID: "8157c822-ab10-4925-b3c9-6211dff26119"). InnerVolumeSpecName "kube-api-access-fhvkt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:05:19.752589 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:19.752575    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8157c822-ab10-4925-b3c9-6211dff26119-maas-api-tls" pod "8157c822-ab10-4925-b3c9-6211dff26119" (UID: "8157c822-ab10-4925-b3c9-6211dff26119"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:05:19.787686 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:19.787651    4477 generic.go:356] "Generic (PLEG): container finished" podID="e8a3aba3-7d06-4fd5-8106-2d02250f1da2" containerID="019a986e536c9e2adbd8f4c7a9976538e38f3215dbb98967f7b5f12eb6167d2a" exitCode=0
Oct 30 23:05:19.787865 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:19.787847    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-cjkc9"
Oct 30 23:05:19.788882 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:19.788578    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-cjkc9" event={"ID":"e8a3aba3-7d06-4fd5-8106-2d02250f1da2","Type":"ContainerDied","Data":"019a986e536c9e2adbd8f4c7a9976538e38f3215dbb98967f7b5f12eb6167d2a"}
Oct 30 23:05:19.788882 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:19.788696    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-cjkc9" event={"ID":"e8a3aba3-7d06-4fd5-8106-2d02250f1da2","Type":"ContainerDied","Data":"7b56f5622e6920d5107afe8f3a0d8c883d59419da5d18ee840fdaa36cbe69eeb"}
Oct 30 23:05:19.788882 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:19.788739    4477 scope.go:122] "RemoveContainer" containerID="019a986e536c9e2adbd8f4c7a9976538e38f3215dbb98967f7b5f12eb6167d2a"
Oct 30 23:05:19.792177 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:19.792152    4477 generic.go:356] "Generic (PLEG): container finished" podID="8157c822-ab10-4925-b3c9-6211dff26119" containerID="c715d01dee2aaf025072e7a92e611d31396fa32e4b4a9a4b55193f32222254e0" exitCode=0
Oct 30 23:05:19.792317 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:19.792301    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e14e00-6bfcb896b7-xq4xg" event={"ID":"8157c822-ab10-4925-b3c9-6211dff26119","Type":"ContainerDied","Data":"c715d01dee2aaf025072e7a92e611d31396fa32e4b4a9a4b55193f32222254e0"}
Oct 30 23:05:19.792370 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:19.792344    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e14e00-6bfcb896b7-xq4xg" event={"ID":"8157c822-ab10-4925-b3c9-6211dff26119","Type":"ContainerDied","Data":"ba5918f47726bbfb75336f28886891e257248325ae530b3be91ce6b85b948027"}
Oct 30 23:05:19.792424 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:19.792412    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e14e00-6bfcb896b7-xq4xg"
Oct 30 23:05:19.797393 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:19.797242    4477 generic.go:356] "Generic (PLEG): container finished" podID="9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7" containerID="ed35cf609bf857a95d6417e89e7792109358a7da9992f72952de20867bf7ce4c" exitCode=0
Oct 30 23:05:19.797393 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:19.797296    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-pmswk"
Oct 30 23:05:19.797393 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:19.797304    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-pmswk" event={"ID":"9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7","Type":"ContainerDied","Data":"ed35cf609bf857a95d6417e89e7792109358a7da9992f72952de20867bf7ce4c"}
Oct 30 23:05:19.797393 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:19.797336    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-pmswk" event={"ID":"9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7","Type":"ContainerDied","Data":"09186d3518dbc9cb555def445fb742dd0b64f68d6ef06d3d871d43690abd1825"}
Oct 30 23:05:19.801518 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:19.801468    4477 scope.go:122] "RemoveContainer" containerID="019a986e536c9e2adbd8f4c7a9976538e38f3215dbb98967f7b5f12eb6167d2a"
Oct 30 23:05:19.801829 ip-10-0-130-77 kubenswrapper[4477]: E1030 23:05:19.801802    4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"019a986e536c9e2adbd8f4c7a9976538e38f3215dbb98967f7b5f12eb6167d2a\": container with ID starting with 019a986e536c9e2adbd8f4c7a9976538e38f3215dbb98967f7b5f12eb6167d2a not found: ID does not exist" containerID="019a986e536c9e2adbd8f4c7a9976538e38f3215dbb98967f7b5f12eb6167d2a"
Oct 30 23:05:19.801901 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:19.801839    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"019a986e536c9e2adbd8f4c7a9976538e38f3215dbb98967f7b5f12eb6167d2a"} err="failed to get container status \"019a986e536c9e2adbd8f4c7a9976538e38f3215dbb98967f7b5f12eb6167d2a\": rpc error: code = NotFound desc = could not find container \"019a986e536c9e2adbd8f4c7a9976538e38f3215dbb98967f7b5f12eb6167d2a\": container with ID starting with 019a986e536c9e2adbd8f4c7a9976538e38f3215dbb98967f7b5f12eb6167d2a not found: ID does not exist"
Oct 30 23:05:19.801901 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:19.801863    4477 scope.go:122] "RemoveContainer" containerID="c715d01dee2aaf025072e7a92e611d31396fa32e4b4a9a4b55193f32222254e0"
Oct 30 23:05:19.812284 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:19.812269    4477 scope.go:122] "RemoveContainer" containerID="c715d01dee2aaf025072e7a92e611d31396fa32e4b4a9a4b55193f32222254e0"
Oct 30 23:05:19.812552 ip-10-0-130-77 kubenswrapper[4477]: E1030 23:05:19.812534    4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c715d01dee2aaf025072e7a92e611d31396fa32e4b4a9a4b55193f32222254e0\": container with ID starting with c715d01dee2aaf025072e7a92e611d31396fa32e4b4a9a4b55193f32222254e0 not found: ID does not exist" containerID="c715d01dee2aaf025072e7a92e611d31396fa32e4b4a9a4b55193f32222254e0"
Oct 30 23:05:19.812620 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:19.812558    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c715d01dee2aaf025072e7a92e611d31396fa32e4b4a9a4b55193f32222254e0"} err="failed to get container status \"c715d01dee2aaf025072e7a92e611d31396fa32e4b4a9a4b55193f32222254e0\": rpc error: code = NotFound desc = could not find container \"c715d01dee2aaf025072e7a92e611d31396fa32e4b4a9a4b55193f32222254e0\": container with ID starting with c715d01dee2aaf025072e7a92e611d31396fa32e4b4a9a4b55193f32222254e0 not found: ID does not exist"
Oct 30 23:05:19.812620 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:19.812573    4477 scope.go:122] "RemoveContainer" containerID="ed35cf609bf857a95d6417e89e7792109358a7da9992f72952de20867bf7ce4c"
Oct 30 23:05:19.820617 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:19.820601    4477 scope.go:122] "RemoveContainer" containerID="ed35cf609bf857a95d6417e89e7792109358a7da9992f72952de20867bf7ce4c"
Oct 30 23:05:19.820872 ip-10-0-130-77 kubenswrapper[4477]: E1030 23:05:19.820855    4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed35cf609bf857a95d6417e89e7792109358a7da9992f72952de20867bf7ce4c\": container with ID starting with ed35cf609bf857a95d6417e89e7792109358a7da9992f72952de20867bf7ce4c not found: ID does not exist" containerID="ed35cf609bf857a95d6417e89e7792109358a7da9992f72952de20867bf7ce4c"
Oct 30 23:05:19.820926 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:19.820876    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed35cf609bf857a95d6417e89e7792109358a7da9992f72952de20867bf7ce4c"} err="failed to get container status \"ed35cf609bf857a95d6417e89e7792109358a7da9992f72952de20867bf7ce4c\": rpc error: code = NotFound desc = could not find container \"ed35cf609bf857a95d6417e89e7792109358a7da9992f72952de20867bf7ce4c\": container with ID starting with ed35cf609bf857a95d6417e89e7792109358a7da9992f72952de20867bf7ce4c not found: ID does not exist"
Oct 30 23:05:19.851318 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:19.851297    4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhvkt\" (UniqueName: \"kubernetes.io/projected/8157c822-ab10-4925-b3c9-6211dff26119-kube-api-access-fhvkt\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:05:19.851318 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:19.851317    4477 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8157c822-ab10-4925-b3c9-6211dff26119-maas-api-tls\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:05:19.851432 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:19.851329    4477 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8157c822-ab10-4925-b3c9-6211dff26119-metrics-certs\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:05:19.862793 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:19.862772    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-pmswk"]
Oct 30 23:05:19.887911 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:19.887889    4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-09d0d5-openshift-default-fdf9b9665-pmswk"]
Oct 30 23:05:19.919294 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:19.919275    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-cjkc9"]
Oct 30 23:05:19.921222 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:19.921204    4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-2b1e88-openshift-default-84bb47bf79-cjkc9"]
Oct 30 23:05:19.952659 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:19.952633    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e14e00-6bfcb896b7-xq4xg"]
Oct 30 23:05:19.956388 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:19.956369    4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e14e00-6bfcb896b7-xq4xg"]
Oct 30 23:05:20.192534 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:20.192501    4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8157c822-ab10-4925-b3c9-6211dff26119" path="/var/lib/kubelet/pods/8157c822-ab10-4925-b3c9-6211dff26119/volumes"
Oct 30 23:05:20.193166 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:20.193147    4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7" path="/var/lib/kubelet/pods/9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7/volumes"
Oct 30 23:05:20.193638 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:20.193618    4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8a3aba3-7d06-4fd5-8106-2d02250f1da2" path="/var/lib/kubelet/pods/e8a3aba3-7d06-4fd5-8106-2d02250f1da2/volumes"
Oct 30 23:05:23.332054 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:23.332007    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6bz92"]
Oct 30 23:05:26.265728 ip-10-0-130-77 kubenswrapper[4477]: E1030 23:05:26.265681    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 23:05:35.065066 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:35.065018    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-d52c59/distinct-w3-d52c59-kserve-ffb5f5b96-l58xl"]
Oct 30 23:05:50.085996 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:50.085953    4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-d52c59/distinct-w3-d52c59-kserve-ffb5f5b96-l58xl" podUID="c7b1483b-2631-4aa2-a888-3096761d5494" containerName="main" containerID="cri-o://a543823ac89b6d62bf8c1e49348e2349a689bbacde460605ed87092c88e9df96" gracePeriod=45
Oct 30 23:05:50.343717 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:50.343661    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d52c59/distinct-w3-d52c59-kserve-ffb5f5b96-l58xl"
Oct 30 23:05:50.434577 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:50.434540    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7b1483b-2631-4aa2-a888-3096761d5494-tls-certs\" (UniqueName: \"kubernetes.io/secret/c7b1483b-2631-4aa2-a888-3096761d5494-tls-certs\") pod \"c7b1483b-2631-4aa2-a888-3096761d5494\" (UID: \"c7b1483b-2631-4aa2-a888-3096761d5494\") "
Oct 30 23:05:50.434740 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:50.434638    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7b1483b-2631-4aa2-a888-3096761d5494-kube-api-access-bsc7p\" (UniqueName: \"kubernetes.io/projected/c7b1483b-2631-4aa2-a888-3096761d5494-kube-api-access-bsc7p\") pod \"c7b1483b-2631-4aa2-a888-3096761d5494\" (UID: \"c7b1483b-2631-4aa2-a888-3096761d5494\") "
Oct 30 23:05:50.434804 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:50.434782    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7b1483b-2631-4aa2-a888-3096761d5494-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7b1483b-2631-4aa2-a888-3096761d5494-model-cache\") pod \"c7b1483b-2631-4aa2-a888-3096761d5494\" (UID: \"c7b1483b-2631-4aa2-a888-3096761d5494\") "
Oct 30 23:05:50.434905 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:50.434880    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7b1483b-2631-4aa2-a888-3096761d5494-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7b1483b-2631-4aa2-a888-3096761d5494-tmp-dir\") pod \"c7b1483b-2631-4aa2-a888-3096761d5494\" (UID: \"c7b1483b-2631-4aa2-a888-3096761d5494\") "
Oct 30 23:05:50.435080 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:50.434923    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7b1483b-2631-4aa2-a888-3096761d5494-home\" (UniqueName: \"kubernetes.io/empty-dir/c7b1483b-2631-4aa2-a888-3096761d5494-home\") pod \"c7b1483b-2631-4aa2-a888-3096761d5494\" (UID: \"c7b1483b-2631-4aa2-a888-3096761d5494\") "
Oct 30 23:05:50.435080 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:50.434984    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7b1483b-2631-4aa2-a888-3096761d5494-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7b1483b-2631-4aa2-a888-3096761d5494-dshm\") pod \"c7b1483b-2631-4aa2-a888-3096761d5494\" (UID: \"c7b1483b-2631-4aa2-a888-3096761d5494\") "
Oct 30 23:05:50.435229 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:50.435086    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7b1483b-2631-4aa2-a888-3096761d5494-model-cache" pod "c7b1483b-2631-4aa2-a888-3096761d5494" (UID: "c7b1483b-2631-4aa2-a888-3096761d5494"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:05:50.435229 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:50.435141    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7b1483b-2631-4aa2-a888-3096761d5494-tmp-dir" pod "c7b1483b-2631-4aa2-a888-3096761d5494" (UID: "c7b1483b-2631-4aa2-a888-3096761d5494"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:05:50.435344 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:50.435331    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7b1483b-2631-4aa2-a888-3096761d5494-home" pod "c7b1483b-2631-4aa2-a888-3096761d5494" (UID: "c7b1483b-2631-4aa2-a888-3096761d5494"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:05:50.435429 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:50.435413    4477 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7b1483b-2631-4aa2-a888-3096761d5494-model-cache\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:05:50.435502 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:50.435431    4477 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7b1483b-2631-4aa2-a888-3096761d5494-tmp-dir\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:05:50.435502 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:50.435441    4477 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7b1483b-2631-4aa2-a888-3096761d5494-home\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:05:50.436991 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:50.436966    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7b1483b-2631-4aa2-a888-3096761d5494-kube-api-access-bsc7p" pod "c7b1483b-2631-4aa2-a888-3096761d5494" (UID: "c7b1483b-2631-4aa2-a888-3096761d5494"). InnerVolumeSpecName "kube-api-access-bsc7p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:05:50.437115 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:50.436971    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7b1483b-2631-4aa2-a888-3096761d5494-tls-certs" pod "c7b1483b-2631-4aa2-a888-3096761d5494" (UID: "c7b1483b-2631-4aa2-a888-3096761d5494"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:05:50.437115 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:50.437024    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7b1483b-2631-4aa2-a888-3096761d5494-dshm" pod "c7b1483b-2631-4aa2-a888-3096761d5494" (UID: "c7b1483b-2631-4aa2-a888-3096761d5494"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:05:50.536413 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:50.536378    4477 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7b1483b-2631-4aa2-a888-3096761d5494-dshm\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:05:50.536413 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:50.536408    4477 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7b1483b-2631-4aa2-a888-3096761d5494-tls-certs\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:05:50.536413 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:50.536419    4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bsc7p\" (UniqueName: \"kubernetes.io/projected/c7b1483b-2631-4aa2-a888-3096761d5494-kube-api-access-bsc7p\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:05:50.965240 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:50.965204    4477 generic.go:356] "Generic (PLEG): container finished" podID="c7b1483b-2631-4aa2-a888-3096761d5494" containerID="a543823ac89b6d62bf8c1e49348e2349a689bbacde460605ed87092c88e9df96" exitCode=0
Oct 30 23:05:50.965440 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:50.965244    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d52c59/distinct-w3-d52c59-kserve-ffb5f5b96-l58xl"
Oct 30 23:05:50.965440 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:50.965284    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d52c59/distinct-w3-d52c59-kserve-ffb5f5b96-l58xl" event={"ID":"c7b1483b-2631-4aa2-a888-3096761d5494","Type":"ContainerDied","Data":"a543823ac89b6d62bf8c1e49348e2349a689bbacde460605ed87092c88e9df96"}
Oct 30 23:05:50.965440 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:50.965322    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d52c59/distinct-w3-d52c59-kserve-ffb5f5b96-l58xl" event={"ID":"c7b1483b-2631-4aa2-a888-3096761d5494","Type":"ContainerDied","Data":"5d2db8ec0feb870a7866ef8138804196aa348c182a4b7c8c1f702ee2e9208bda"}
Oct 30 23:05:50.965440 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:50.965338    4477 scope.go:122] "RemoveContainer" containerID="a543823ac89b6d62bf8c1e49348e2349a689bbacde460605ed87092c88e9df96"
Oct 30 23:05:50.975076 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:50.975059    4477 scope.go:122] "RemoveContainer" containerID="a543823ac89b6d62bf8c1e49348e2349a689bbacde460605ed87092c88e9df96"
Oct 30 23:05:50.975338 ip-10-0-130-77 kubenswrapper[4477]: E1030 23:05:50.975322    4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a543823ac89b6d62bf8c1e49348e2349a689bbacde460605ed87092c88e9df96\": container with ID starting with a543823ac89b6d62bf8c1e49348e2349a689bbacde460605ed87092c88e9df96 not found: ID does not exist" containerID="a543823ac89b6d62bf8c1e49348e2349a689bbacde460605ed87092c88e9df96"
Oct 30 23:05:50.975391 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:50.975346    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a543823ac89b6d62bf8c1e49348e2349a689bbacde460605ed87092c88e9df96"} err="failed to get container status \"a543823ac89b6d62bf8c1e49348e2349a689bbacde460605ed87092c88e9df96\": rpc error: code = NotFound desc = could not find container \"a543823ac89b6d62bf8c1e49348e2349a689bbacde460605ed87092c88e9df96\": container with ID starting with a543823ac89b6d62bf8c1e49348e2349a689bbacde460605ed87092c88e9df96 not found: ID does not exist"
Oct 30 23:05:51.012125 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:51.012004    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-d52c59/distinct-w3-d52c59-kserve-ffb5f5b96-l58xl"]
Oct 30 23:05:51.022815 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:51.022792    4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-d52c59/distinct-w3-d52c59-kserve-ffb5f5b96-l58xl"]
Oct 30 23:05:52.193321 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:52.193292    4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7b1483b-2631-4aa2-a888-3096761d5494" path="/var/lib/kubelet/pods/c7b1483b-2631-4aa2-a888-3096761d5494/volumes"
Oct 30 23:05:55.283150 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:55.283118    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6bz92"]
Oct 30 23:05:56.265801 ip-10-0-130-77 kubenswrapper[4477]: E1030 23:05:56.265765    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 23:05:57.488931 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:57.488899    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-d52c59-bbfdfb8dd-d7cx7"]
Oct 30 23:05:57.489446 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:57.489111    4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d52c59-bbfdfb8dd-d7cx7" podUID="dd421f77-d9c7-4bb8-833a-e9a861fa468f" containerName="maas-api" containerID="cri-o://22ea87c9a8551f2c9dadaadb980faf2ef7f458fd368e7ef903cbdbfc37fb6b0f" gracePeriod=30
Oct 30 23:05:57.755439 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:57.755372    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d52c59-bbfdfb8dd-d7cx7"
Oct 30 23:05:57.798758 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:57.798724    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd421f77-d9c7-4bb8-833a-e9a861fa468f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd421f77-d9c7-4bb8-833a-e9a861fa468f-maas-api-tls\") pod \"dd421f77-d9c7-4bb8-833a-e9a861fa468f\" (UID: \"dd421f77-d9c7-4bb8-833a-e9a861fa468f\") "
Oct 30 23:05:57.798951 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:57.798786    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd421f77-d9c7-4bb8-833a-e9a861fa468f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd421f77-d9c7-4bb8-833a-e9a861fa468f-openshift-service-ca\") pod \"dd421f77-d9c7-4bb8-833a-e9a861fa468f\" (UID: \"dd421f77-d9c7-4bb8-833a-e9a861fa468f\") "
Oct 30 23:05:57.798951 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:57.798813    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd421f77-d9c7-4bb8-833a-e9a861fa468f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd421f77-d9c7-4bb8-833a-e9a861fa468f-metrics-certs\") pod \"dd421f77-d9c7-4bb8-833a-e9a861fa468f\" (UID: \"dd421f77-d9c7-4bb8-833a-e9a861fa468f\") "
Oct 30 23:05:57.799113 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:57.798959    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd421f77-d9c7-4bb8-833a-e9a861fa468f-kube-api-access-dxxkf\" (UniqueName: \"kubernetes.io/projected/dd421f77-d9c7-4bb8-833a-e9a861fa468f-kube-api-access-dxxkf\") pod \"dd421f77-d9c7-4bb8-833a-e9a861fa468f\" (UID: \"dd421f77-d9c7-4bb8-833a-e9a861fa468f\") "
Oct 30 23:05:57.799240 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:57.799211    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd421f77-d9c7-4bb8-833a-e9a861fa468f-openshift-service-ca" pod "dd421f77-d9c7-4bb8-833a-e9a861fa468f" (UID: "dd421f77-d9c7-4bb8-833a-e9a861fa468f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:05:57.799403 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:57.799386    4477 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd421f77-d9c7-4bb8-833a-e9a861fa468f-openshift-service-ca\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:05:57.801398 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:57.801372    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd421f77-d9c7-4bb8-833a-e9a861fa468f-metrics-certs" pod "dd421f77-d9c7-4bb8-833a-e9a861fa468f" (UID: "dd421f77-d9c7-4bb8-833a-e9a861fa468f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:05:57.802984 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:57.802691    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd421f77-d9c7-4bb8-833a-e9a861fa468f-kube-api-access-dxxkf" pod "dd421f77-d9c7-4bb8-833a-e9a861fa468f" (UID: "dd421f77-d9c7-4bb8-833a-e9a861fa468f"). InnerVolumeSpecName "kube-api-access-dxxkf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:05:57.802984 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:57.802923    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd421f77-d9c7-4bb8-833a-e9a861fa468f-maas-api-tls" pod "dd421f77-d9c7-4bb8-833a-e9a861fa468f" (UID: "dd421f77-d9c7-4bb8-833a-e9a861fa468f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:05:57.900935 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:57.900894    4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxxkf\" (UniqueName: \"kubernetes.io/projected/dd421f77-d9c7-4bb8-833a-e9a861fa468f-kube-api-access-dxxkf\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:05:57.901092 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:57.900937    4477 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd421f77-d9c7-4bb8-833a-e9a861fa468f-maas-api-tls\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:05:57.901092 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:57.900953    4477 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd421f77-d9c7-4bb8-833a-e9a861fa468f-metrics-certs\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:05:58.003274 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:58.003235    4477 generic.go:356] "Generic (PLEG): container finished" podID="dd421f77-d9c7-4bb8-833a-e9a861fa468f" containerID="22ea87c9a8551f2c9dadaadb980faf2ef7f458fd368e7ef903cbdbfc37fb6b0f" exitCode=0
Oct 30 23:05:58.003432 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:58.003288    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d52c59-bbfdfb8dd-d7cx7"
Oct 30 23:05:58.003432 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:58.003323    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d52c59-bbfdfb8dd-d7cx7" event={"ID":"dd421f77-d9c7-4bb8-833a-e9a861fa468f","Type":"ContainerDied","Data":"22ea87c9a8551f2c9dadaadb980faf2ef7f458fd368e7ef903cbdbfc37fb6b0f"}
Oct 30 23:05:58.003432 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:58.003375    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-d52c59-bbfdfb8dd-d7cx7" event={"ID":"dd421f77-d9c7-4bb8-833a-e9a861fa468f","Type":"ContainerDied","Data":"682490b0adfa5bc278e84fe5be9b87d6e17ca83d362adca264d0506f210af21d"}
Oct 30 23:05:58.003432 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:58.003400    4477 scope.go:122] "RemoveContainer" containerID="22ea87c9a8551f2c9dadaadb980faf2ef7f458fd368e7ef903cbdbfc37fb6b0f"
Oct 30 23:05:58.012848 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:58.012831    4477 scope.go:122] "RemoveContainer" containerID="22ea87c9a8551f2c9dadaadb980faf2ef7f458fd368e7ef903cbdbfc37fb6b0f"
Oct 30 23:05:58.013217 ip-10-0-130-77 kubenswrapper[4477]: E1030 23:05:58.013199    4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22ea87c9a8551f2c9dadaadb980faf2ef7f458fd368e7ef903cbdbfc37fb6b0f\": container with ID starting with 22ea87c9a8551f2c9dadaadb980faf2ef7f458fd368e7ef903cbdbfc37fb6b0f not found: ID does not exist" containerID="22ea87c9a8551f2c9dadaadb980faf2ef7f458fd368e7ef903cbdbfc37fb6b0f"
Oct 30 23:05:58.013284 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:58.013226    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22ea87c9a8551f2c9dadaadb980faf2ef7f458fd368e7ef903cbdbfc37fb6b0f"} err="failed to get container status \"22ea87c9a8551f2c9dadaadb980faf2ef7f458fd368e7ef903cbdbfc37fb6b0f\": rpc error: code = NotFound desc = could not find container \"22ea87c9a8551f2c9dadaadb980faf2ef7f458fd368e7ef903cbdbfc37fb6b0f\": container with ID starting with 22ea87c9a8551f2c9dadaadb980faf2ef7f458fd368e7ef903cbdbfc37fb6b0f not found: ID does not exist"
Oct 30 23:05:58.048756 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:58.048728    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-d52c59-bbfdfb8dd-d7cx7"]
Oct 30 23:05:58.052759 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:58.052737    4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-d52c59-bbfdfb8dd-d7cx7"]
Oct 30 23:05:58.199415 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:05:58.199368    4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd421f77-d9c7-4bb8-833a-e9a861fa468f" path="/var/lib/kubelet/pods/dd421f77-d9c7-4bb8-833a-e9a861fa468f/volumes"
Oct 30 23:06:04.684180 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:06:04.684143    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6bz92"]
Oct 30 23:06:05.630063 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:06:05.622837    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-tbj69"]
Oct 30 23:06:05.630063 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:06:05.623165    4477 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-tbj69" podUID="f5d9c5bf-3e60-40b7-9dc0-3588b27074b8" containerName="istio-proxy" containerID="cri-o://b302ec196a15dbe48c2859dbad3a0f514c1fbb086e51a0467efcba706bc26534" gracePeriod=30
Oct 30 23:06:10.890605 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:06:10.890580    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-tbj69"
Oct 30 23:06:11.021606 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:06:11.021525    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5d9c5bf-3e60-40b7-9dc0-3588b27074b8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5d9c5bf-3e60-40b7-9dc0-3588b27074b8-istiod-ca-cert\") pod \"f5d9c5bf-3e60-40b7-9dc0-3588b27074b8\" (UID: \"f5d9c5bf-3e60-40b7-9dc0-3588b27074b8\") "
Oct 30 23:06:11.021606 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:06:11.021586    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5d9c5bf-3e60-40b7-9dc0-3588b27074b8-istio-token\" (UniqueName: \"kubernetes.io/projected/f5d9c5bf-3e60-40b7-9dc0-3588b27074b8-istio-token\") pod \"f5d9c5bf-3e60-40b7-9dc0-3588b27074b8\" (UID: \"f5d9c5bf-3e60-40b7-9dc0-3588b27074b8\") "
Oct 30 23:06:11.021838 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:06:11.021622    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f5d9c5bf-3e60-40b7-9dc0-3588b27074b8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5d9c5bf-3e60-40b7-9dc0-3588b27074b8-istio-podinfo\") pod \"f5d9c5bf-3e60-40b7-9dc0-3588b27074b8\" (UID: \"f5d9c5bf-3e60-40b7-9dc0-3588b27074b8\") "
Oct 30 23:06:11.021838 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:06:11.021694    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5d9c5bf-3e60-40b7-9dc0-3588b27074b8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5d9c5bf-3e60-40b7-9dc0-3588b27074b8-istio-envoy\") pod \"f5d9c5bf-3e60-40b7-9dc0-3588b27074b8\" (UID: \"f5d9c5bf-3e60-40b7-9dc0-3588b27074b8\") "
Oct 30 23:06:11.021838 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:06:11.021731    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5d9c5bf-3e60-40b7-9dc0-3588b27074b8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5d9c5bf-3e60-40b7-9dc0-3588b27074b8-workload-certs\") pod \"f5d9c5bf-3e60-40b7-9dc0-3588b27074b8\" (UID: \"f5d9c5bf-3e60-40b7-9dc0-3588b27074b8\") "
Oct 30 23:06:11.021838 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:06:11.021773    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5d9c5bf-3e60-40b7-9dc0-3588b27074b8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5d9c5bf-3e60-40b7-9dc0-3588b27074b8-credential-socket\") pod \"f5d9c5bf-3e60-40b7-9dc0-3588b27074b8\" (UID: \"f5d9c5bf-3e60-40b7-9dc0-3588b27074b8\") "
Oct 30 23:06:11.022088 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:06:11.021931    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5d9c5bf-3e60-40b7-9dc0-3588b27074b8-istiod-ca-cert" pod "f5d9c5bf-3e60-40b7-9dc0-3588b27074b8" (UID: "f5d9c5bf-3e60-40b7-9dc0-3588b27074b8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:06:11.022088 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:06:11.021988    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5d9c5bf-3e60-40b7-9dc0-3588b27074b8-kube-api-access-w4s27\" (UniqueName: \"kubernetes.io/projected/f5d9c5bf-3e60-40b7-9dc0-3588b27074b8-kube-api-access-w4s27\") pod \"f5d9c5bf-3e60-40b7-9dc0-3588b27074b8\" (UID: \"f5d9c5bf-3e60-40b7-9dc0-3588b27074b8\") "
Oct 30 23:06:11.022220 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:06:11.022175    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5d9c5bf-3e60-40b7-9dc0-3588b27074b8-credential-socket" pod "f5d9c5bf-3e60-40b7-9dc0-3588b27074b8" (UID: "f5d9c5bf-3e60-40b7-9dc0-3588b27074b8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:06:11.022220 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:06:11.022073    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5d9c5bf-3e60-40b7-9dc0-3588b27074b8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5d9c5bf-3e60-40b7-9dc0-3588b27074b8-istio-data\") pod \"f5d9c5bf-3e60-40b7-9dc0-3588b27074b8\" (UID: \"f5d9c5bf-3e60-40b7-9dc0-3588b27074b8\") "
Oct 30 23:06:11.022329 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:06:11.022269    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5d9c5bf-3e60-40b7-9dc0-3588b27074b8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5d9c5bf-3e60-40b7-9dc0-3588b27074b8-workload-socket\") pod \"f5d9c5bf-3e60-40b7-9dc0-3588b27074b8\" (UID: \"f5d9c5bf-3e60-40b7-9dc0-3588b27074b8\") "
Oct 30 23:06:11.022385 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:06:11.022343    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5d9c5bf-3e60-40b7-9dc0-3588b27074b8-istio-data" pod "f5d9c5bf-3e60-40b7-9dc0-3588b27074b8" (UID: "f5d9c5bf-3e60-40b7-9dc0-3588b27074b8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:06:11.022442 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:06:11.022386    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5d9c5bf-3e60-40b7-9dc0-3588b27074b8-workload-certs" pod "f5d9c5bf-3e60-40b7-9dc0-3588b27074b8" (UID: "f5d9c5bf-3e60-40b7-9dc0-3588b27074b8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:06:11.022658 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:06:11.022634    4477 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5d9c5bf-3e60-40b7-9dc0-3588b27074b8-workload-certs\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:06:11.022658 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:06:11.022644    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5d9c5bf-3e60-40b7-9dc0-3588b27074b8-workload-socket" pod "f5d9c5bf-3e60-40b7-9dc0-3588b27074b8" (UID: "f5d9c5bf-3e60-40b7-9dc0-3588b27074b8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:06:11.022829 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:06:11.022661    4477 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5d9c5bf-3e60-40b7-9dc0-3588b27074b8-credential-socket\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:06:11.022829 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:06:11.022675    4477 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5d9c5bf-3e60-40b7-9dc0-3588b27074b8-istio-data\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:06:11.022829 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:06:11.022688    4477 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5d9c5bf-3e60-40b7-9dc0-3588b27074b8-istiod-ca-cert\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:06:11.024004 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:06:11.023972    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5d9c5bf-3e60-40b7-9dc0-3588b27074b8-istio-token" pod "f5d9c5bf-3e60-40b7-9dc0-3588b27074b8" (UID: "f5d9c5bf-3e60-40b7-9dc0-3588b27074b8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:06:11.024230 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:06:11.024213    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5d9c5bf-3e60-40b7-9dc0-3588b27074b8-istio-envoy" pod "f5d9c5bf-3e60-40b7-9dc0-3588b27074b8" (UID: "f5d9c5bf-3e60-40b7-9dc0-3588b27074b8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:06:11.024340 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:06:11.024325    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5d9c5bf-3e60-40b7-9dc0-3588b27074b8-kube-api-access-w4s27" pod "f5d9c5bf-3e60-40b7-9dc0-3588b27074b8" (UID: "f5d9c5bf-3e60-40b7-9dc0-3588b27074b8"). InnerVolumeSpecName "kube-api-access-w4s27". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:06:11.024418 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:06:11.024402    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f5d9c5bf-3e60-40b7-9dc0-3588b27074b8-istio-podinfo" pod "f5d9c5bf-3e60-40b7-9dc0-3588b27074b8" (UID: "f5d9c5bf-3e60-40b7-9dc0-3588b27074b8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:06:11.071557 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:06:11.071522    4477 generic.go:356] "Generic (PLEG): container finished" podID="f5d9c5bf-3e60-40b7-9dc0-3588b27074b8" containerID="b302ec196a15dbe48c2859dbad3a0f514c1fbb086e51a0467efcba706bc26534" exitCode=0
Oct 30 23:06:11.071727 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:06:11.071589    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-tbj69"
Oct 30 23:06:11.071727 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:06:11.071604    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-tbj69" event={"ID":"f5d9c5bf-3e60-40b7-9dc0-3588b27074b8","Type":"ContainerDied","Data":"b302ec196a15dbe48c2859dbad3a0f514c1fbb086e51a0467efcba706bc26534"}
Oct 30 23:06:11.071727 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:06:11.071672    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-tbj69" event={"ID":"f5d9c5bf-3e60-40b7-9dc0-3588b27074b8","Type":"ContainerDied","Data":"379406bf0be7800f8cd5edceb52a4be4603bd1b7883ea8554ecf120bc869a342"}
Oct 30 23:06:11.071727 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:06:11.071701    4477 scope.go:122] "RemoveContainer" containerID="b302ec196a15dbe48c2859dbad3a0f514c1fbb086e51a0467efcba706bc26534"
Oct 30 23:06:11.083863 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:06:11.083767    4477 scope.go:122] "RemoveContainer" containerID="b302ec196a15dbe48c2859dbad3a0f514c1fbb086e51a0467efcba706bc26534"
Oct 30 23:06:11.084352 ip-10-0-130-77 kubenswrapper[4477]: E1030 23:06:11.084327    4477 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b302ec196a15dbe48c2859dbad3a0f514c1fbb086e51a0467efcba706bc26534\": container with ID starting with b302ec196a15dbe48c2859dbad3a0f514c1fbb086e51a0467efcba706bc26534 not found: ID does not exist" containerID="b302ec196a15dbe48c2859dbad3a0f514c1fbb086e51a0467efcba706bc26534"
Oct 30 23:06:11.084467 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:06:11.084363    4477 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b302ec196a15dbe48c2859dbad3a0f514c1fbb086e51a0467efcba706bc26534"} err="failed to get container status \"b302ec196a15dbe48c2859dbad3a0f514c1fbb086e51a0467efcba706bc26534\": rpc error: code = NotFound desc = could not find container \"b302ec196a15dbe48c2859dbad3a0f514c1fbb086e51a0467efcba706bc26534\": container with ID starting with b302ec196a15dbe48c2859dbad3a0f514c1fbb086e51a0467efcba706bc26534 not found: ID does not exist"
Oct 30 23:06:11.120307 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:06:11.120279    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-tbj69"]
Oct 30 23:06:11.123166 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:06:11.123139    4477 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5d9c5bf-3e60-40b7-9dc0-3588b27074b8-istio-envoy\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:06:11.123166 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:06:11.123164    4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4s27\" (UniqueName: \"kubernetes.io/projected/f5d9c5bf-3e60-40b7-9dc0-3588b27074b8-kube-api-access-w4s27\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:06:11.123404 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:06:11.123174    4477 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5d9c5bf-3e60-40b7-9dc0-3588b27074b8-workload-socket\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:06:11.123404 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:06:11.123185    4477 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5d9c5bf-3e60-40b7-9dc0-3588b27074b8-istio-token\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:06:11.123404 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:06:11.123199    4477 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5d9c5bf-3e60-40b7-9dc0-3588b27074b8-istio-podinfo\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:06:11.123539 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:06:11.123430    4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-d52c59-openshift-default-5466968fb7-tbj69"]
Oct 30 23:06:12.195486 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:06:12.195453    4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5d9c5bf-3e60-40b7-9dc0-3588b27074b8" path="/var/lib/kubelet/pods/f5d9c5bf-3e60-40b7-9dc0-3588b27074b8/volumes"
Oct 30 23:06:26.265459 ip-10-0-130-77 kubenswrapper[4477]: E1030 23:06:26.265419    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 23:06:56.264802 ip-10-0-130-77 kubenswrapper[4477]: E1030 23:06:56.264767    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 23:06:57.155868 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:06:57.155842    4477 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jcf8l_015f218d-bf1f-4e7e-bb5d-b805af003246/ovn-acl-logging/0.log"
Oct 30 23:06:57.158395 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:06:57.158371    4477 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jcf8l_015f218d-bf1f-4e7e-bb5d-b805af003246/ovn-acl-logging/0.log"
Oct 30 23:07:25.284036 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:25.283991    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6bz92"]
Oct 30 23:07:26.266398 ip-10-0-130-77 kubenswrapper[4477]: E1030 23:07:26.266356    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 23:07:42.324071 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:42.323839    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6bz92"]
Oct 30 23:07:44.374738 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:44.374696    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-d8sgs"]
Oct 30 23:07:44.376173 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:44.376121    4477 cpu_manager.go:414] "containerMap: removing container" podUID="61990d22-ab8f-45bc-a4d6-4c8d528742f1" containerName="main"
Oct 30 23:07:44.376428 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:44.376405    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61990d22-ab8f-45bc-a4d6-4c8d528742f1" containerName="main"
Oct 30 23:07:44.376562 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:44.376542    4477 cpu_manager.go:414] "containerMap: removing container" podUID="8326b014-989b-4722-befb-6fa8ea60688b" containerName="main"
Oct 30 23:07:44.376562 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:44.376558    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8326b014-989b-4722-befb-6fa8ea60688b" containerName="main"
Oct 30 23:07:44.376800 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:44.376578    4477 cpu_manager.go:414] "containerMap: removing container" podUID="763c695b-9f27-48e4-80a6-ae3c22bc9816" containerName="maas-api"
Oct 30 23:07:44.376800 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:44.376585    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="763c695b-9f27-48e4-80a6-ae3c22bc9816" containerName="maas-api"
Oct 30 23:07:44.376800 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:44.376601    4477 cpu_manager.go:414] "containerMap: removing container" podUID="00783ce5-3f4a-4649-9c93-28b8a1c01b1e" containerName="maas-api"
Oct 30 23:07:44.376800 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:44.376609    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00783ce5-3f4a-4649-9c93-28b8a1c01b1e" containerName="maas-api"
Oct 30 23:07:44.376800 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:44.376627    4477 cpu_manager.go:414] "containerMap: removing container" podUID="cdcfe2c6-edf0-42a3-ac07-434812bedaa5" containerName="cleanup"
Oct 30 23:07:44.376800 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:44.376640    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdcfe2c6-edf0-42a3-ac07-434812bedaa5" containerName="cleanup"
Oct 30 23:07:44.376800 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:44.376651    4477 cpu_manager.go:414] "containerMap: removing container" podUID="a637e6be-489f-4d98-bc02-bca751282d13" containerName="maas-api"
Oct 30 23:07:44.376800 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:44.376663    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a637e6be-489f-4d98-bc02-bca751282d13" containerName="maas-api"
Oct 30 23:07:44.376800 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:44.376686    4477 cpu_manager.go:414] "containerMap: removing container" podUID="ec4328f4-72a8-4462-b87c-66415cbc6989" containerName="maas-api"
Oct 30 23:07:44.376800 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:44.376695    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec4328f4-72a8-4462-b87c-66415cbc6989" containerName="maas-api"
Oct 30 23:07:44.376800 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:44.376706    4477 cpu_manager.go:414] "containerMap: removing container" podUID="401f33fa-f0da-4809-af95-3f4d30ebc830" containerName="maas-api"
Oct 30 23:07:44.376800 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:44.376712    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="401f33fa-f0da-4809-af95-3f4d30ebc830" containerName="maas-api"
Oct 30 23:07:44.376800 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:44.376736    4477 cpu_manager.go:414] "containerMap: removing container" podUID="c67bfb3e-cc4c-40a2-9d8f-bc36f48f2a2f" containerName="maas-api"
Oct 30 23:07:44.376800 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:44.376745    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c67bfb3e-cc4c-40a2-9d8f-bc36f48f2a2f" containerName="maas-api"
Oct 30 23:07:44.376800 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:44.376763    4477 cpu_manager.go:414] "containerMap: removing container" podUID="f5d9c5bf-3e60-40b7-9dc0-3588b27074b8" containerName="istio-proxy"
Oct 30 23:07:44.376800 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:44.376770    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5d9c5bf-3e60-40b7-9dc0-3588b27074b8" containerName="istio-proxy"
Oct 30 23:07:44.376800 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:44.376781    4477 cpu_manager.go:414] "containerMap: removing container" podUID="a910f2eb-f5b8-45fc-a827-103d116454f9" containerName="cleanup"
Oct 30 23:07:44.376800 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:44.376788    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a910f2eb-f5b8-45fc-a827-103d116454f9" containerName="cleanup"
Oct 30 23:07:44.378063 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:44.376814    4477 cpu_manager.go:414] "containerMap: removing container" podUID="30fed99f-b5e6-4c12-a1f4-8d84ed15971e" containerName="istio-proxy"
Oct 30 23:07:44.378063 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:44.376821    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30fed99f-b5e6-4c12-a1f4-8d84ed15971e" containerName="istio-proxy"
Oct 30 23:07:44.378063 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:44.376842    4477 cpu_manager.go:414] "containerMap: removing container" podUID="085847e5-4f50-419e-984f-b612b104ed90" containerName="main"
Oct 30 23:07:44.378063 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:44.376849    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="085847e5-4f50-419e-984f-b612b104ed90" containerName="main"
Oct 30 23:07:44.378063 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:44.376864    4477 cpu_manager.go:414] "containerMap: removing container" podUID="9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7" containerName="istio-proxy"
Oct 30 23:07:44.378063 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:44.376871    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7" containerName="istio-proxy"
Oct 30 23:07:44.378063 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:44.376902    4477 cpu_manager.go:414] "containerMap: removing container" podUID="c17cee11-e2ac-469a-8759-c595e2e73191" containerName="maas-api"
Oct 30 23:07:44.378063 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:44.376909    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c17cee11-e2ac-469a-8759-c595e2e73191" containerName="maas-api"
Oct 30 23:07:44.378063 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:44.376922    4477 cpu_manager.go:414] "containerMap: removing container" podUID="d12a3703-b587-4f7b-9769-d4322c840eeb" containerName="maas-api"
Oct 30 23:07:44.378063 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:44.376929    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d12a3703-b587-4f7b-9769-d4322c840eeb" containerName="maas-api"
Oct 30 23:07:44.378063 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:44.376945    4477 cpu_manager.go:414] "containerMap: removing container" podUID="18a561b5-289d-4401-8377-0be0130d658d" containerName="istio-proxy"
Oct 30 23:07:44.378063 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:44.376952    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18a561b5-289d-4401-8377-0be0130d658d" containerName="istio-proxy"
Oct 30 23:07:44.378063 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:44.376968    4477 cpu_manager.go:414] "containerMap: removing container" podUID="8cbd04b4-bb3e-4bbb-a41a-46131c9de051" containerName="main"
Oct 30 23:07:44.378063 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:44.376975    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cbd04b4-bb3e-4bbb-a41a-46131c9de051" containerName="main"
Oct 30 23:07:44.378063 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:44.376983    4477 cpu_manager.go:414] "containerMap: removing container" podUID="0ec852a7-f59e-4475-817a-fb57174a901d" containerName="maas-api"
Oct 30 23:07:44.378063 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:44.376990    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ec852a7-f59e-4475-817a-fb57174a901d" containerName="maas-api"
Oct 30 23:07:44.378063 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:44.377006    4477 cpu_manager.go:414] "containerMap: removing container" podUID="130008a5-8f15-4ded-b8bd-d1903cb436b2" containerName="maas-api"
Oct 30 23:07:44.378063 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:44.377013    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="130008a5-8f15-4ded-b8bd-d1903cb436b2" containerName="maas-api"
Oct 30 23:07:44.378063 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:44.377021    4477 cpu_manager.go:414] "containerMap: removing container" podUID="738731d9-e739-4c3e-8936-9c4d01f5fc04" containerName="maas-api"
Oct 30 23:07:44.378063 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:44.377028    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="738731d9-e739-4c3e-8936-9c4d01f5fc04" containerName="maas-api"
Oct 30 23:07:44.378063 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:44.377061    4477 cpu_manager.go:414] "containerMap: removing container" podUID="c7b1483b-2631-4aa2-a888-3096761d5494" containerName="main"
Oct 30 23:07:44.378063 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:44.377068    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7b1483b-2631-4aa2-a888-3096761d5494" containerName="main"
Oct 30 23:07:44.378063 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:44.377089    4477 cpu_manager.go:414] "containerMap: removing container" podUID="722f10d8-b7a2-4bf7-a274-9669529ef427" containerName="main"
Oct 30 23:07:44.378063 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:44.377095    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="722f10d8-b7a2-4bf7-a274-9669529ef427" containerName="main"
Oct 30 23:07:44.378063 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:44.377116    4477 cpu_manager.go:414] "containerMap: removing container" podUID="510ad10d-0554-47c8-9b55-deb359795071" containerName="main"
Oct 30 23:07:44.378063 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:44.377122    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="510ad10d-0554-47c8-9b55-deb359795071" containerName="main"
Oct 30 23:07:44.378063 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:44.377131    4477 cpu_manager.go:414] "containerMap: removing container" podUID="e8a3aba3-7d06-4fd5-8106-2d02250f1da2" containerName="istio-proxy"
Oct 30 23:07:44.378063 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:44.377139    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8a3aba3-7d06-4fd5-8106-2d02250f1da2" containerName="istio-proxy"
Oct 30 23:07:44.378063 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:44.377150    4477 cpu_manager.go:414] "containerMap: removing container" podUID="2d8ff9bb-aa7c-46f4-8ff8-d85ba96870a4" containerName="maas-api"
Oct 30 23:07:44.378063 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:44.377157    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d8ff9bb-aa7c-46f4-8ff8-d85ba96870a4" containerName="maas-api"
Oct 30 23:07:44.378063 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:44.377173    4477 cpu_manager.go:414] "containerMap: removing container" podUID="dd421f77-d9c7-4bb8-833a-e9a861fa468f" containerName="maas-api"
Oct 30 23:07:44.378063 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:44.377180    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd421f77-d9c7-4bb8-833a-e9a861fa468f" containerName="maas-api"
Oct 30 23:07:44.378063 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:44.377190    4477 cpu_manager.go:414] "containerMap: removing container" podUID="8157c822-ab10-4925-b3c9-6211dff26119" containerName="maas-api"
Oct 30 23:07:44.378063 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:44.377204    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8157c822-ab10-4925-b3c9-6211dff26119" containerName="maas-api"
Oct 30 23:07:44.378063 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:44.377375    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cbd04b4-bb3e-4bbb-a41a-46131c9de051" containerName="main"
Oct 30 23:07:44.378063 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:44.377396    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ec852a7-f59e-4475-817a-fb57174a901d" containerName="maas-api"
Oct 30 23:07:44.378063 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:44.377405    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="130008a5-8f15-4ded-b8bd-d1903cb436b2" containerName="maas-api"
Oct 30 23:07:44.378063 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:44.377419    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="738731d9-e739-4c3e-8936-9c4d01f5fc04" containerName="maas-api"
Oct 30 23:07:44.378063 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:44.377435    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7b1483b-2631-4aa2-a888-3096761d5494" containerName="main"
Oct 30 23:07:44.378063 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:44.377445    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="722f10d8-b7a2-4bf7-a274-9669529ef427" containerName="main"
Oct 30 23:07:44.378063 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:44.377459    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="510ad10d-0554-47c8-9b55-deb359795071" containerName="main"
Oct 30 23:07:44.378063 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:44.377466    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8a3aba3-7d06-4fd5-8106-2d02250f1da2" containerName="istio-proxy"
Oct 30 23:07:44.378063 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:44.377481    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d8ff9bb-aa7c-46f4-8ff8-d85ba96870a4" containerName="maas-api"
Oct 30 23:07:44.378063 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:44.377490    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd421f77-d9c7-4bb8-833a-e9a861fa468f" containerName="maas-api"
Oct 30 23:07:44.378063 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:44.377505    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="8157c822-ab10-4925-b3c9-6211dff26119" containerName="maas-api"
Oct 30 23:07:44.378063 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:44.377519    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="61990d22-ab8f-45bc-a4d6-4c8d528742f1" containerName="main"
Oct 30 23:07:44.378063 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:44.377528    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="8326b014-989b-4722-befb-6fa8ea60688b" containerName="main"
Oct 30 23:07:44.378063 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:44.377544    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="763c695b-9f27-48e4-80a6-ae3c22bc9816" containerName="maas-api"
Oct 30 23:07:44.378063 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:44.377551    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="00783ce5-3f4a-4649-9c93-28b8a1c01b1e" containerName="maas-api"
Oct 30 23:07:44.378063 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:44.377565    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdcfe2c6-edf0-42a3-ac07-434812bedaa5" containerName="cleanup"
Oct 30 23:07:44.378063 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:44.377572    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="a637e6be-489f-4d98-bc02-bca751282d13" containerName="maas-api"
Oct 30 23:07:44.378063 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:44.377580    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec4328f4-72a8-4462-b87c-66415cbc6989" containerName="maas-api"
Oct 30 23:07:44.378063 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:44.377595    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="401f33fa-f0da-4809-af95-3f4d30ebc830" containerName="maas-api"
Oct 30 23:07:44.378063 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:44.377606    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="c67bfb3e-cc4c-40a2-9d8f-bc36f48f2a2f" containerName="maas-api"
Oct 30 23:07:44.378063 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:44.377622    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5d9c5bf-3e60-40b7-9dc0-3588b27074b8" containerName="istio-proxy"
Oct 30 23:07:44.378063 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:44.377637    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="a910f2eb-f5b8-45fc-a827-103d116454f9" containerName="cleanup"
Oct 30 23:07:44.378063 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:44.377644    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="30fed99f-b5e6-4c12-a1f4-8d84ed15971e" containerName="istio-proxy"
Oct 30 23:07:44.378063 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:44.377660    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="085847e5-4f50-419e-984f-b612b104ed90" containerName="main"
Oct 30 23:07:44.378063 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:44.377678    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="c17cee11-e2ac-469a-8759-c595e2e73191" containerName="maas-api"
Oct 30 23:07:44.378063 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:44.377689    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c1776ee-cba2-46bd-b9ed-1cf9fd5903d7" containerName="istio-proxy"
Oct 30 23:07:44.378063 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:44.377704    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="d12a3703-b587-4f7b-9769-d4322c840eeb" containerName="maas-api"
Oct 30 23:07:44.378063 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:44.377712    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="18a561b5-289d-4401-8377-0be0130d658d" containerName="istio-proxy"
Oct 30 23:07:44.381966 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:44.381943    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-d8sgs"
Oct 30 23:07:44.402108 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:44.402071    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-d8sgs"]
Oct 30 23:07:44.464323 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:44.464293    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d8945b9-b167-4397-bb13-a9ef77e413f7-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-d8sgs\" (UID: \"4d8945b9-b167-4397-bb13-a9ef77e413f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-d8sgs"
Oct 30 23:07:44.464459 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:44.464343    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d8945b9-b167-4397-bb13-a9ef77e413f7-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-d8sgs\" (UID: \"4d8945b9-b167-4397-bb13-a9ef77e413f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-d8sgs"
Oct 30 23:07:44.464459 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:44.464419    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d8945b9-b167-4397-bb13-a9ef77e413f7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-d8sgs\" (UID: \"4d8945b9-b167-4397-bb13-a9ef77e413f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-d8sgs"
Oct 30 23:07:44.464583 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:44.464457    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d8945b9-b167-4397-bb13-a9ef77e413f7-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-d8sgs\" (UID: \"4d8945b9-b167-4397-bb13-a9ef77e413f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-d8sgs"
Oct 30 23:07:44.464583 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:44.464485    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkdhg\" (UniqueName: \"kubernetes.io/projected/4d8945b9-b167-4397-bb13-a9ef77e413f7-kube-api-access-jkdhg\") pod \"maas-default-gateway-openshift-default-6db586dd6c-d8sgs\" (UID: \"4d8945b9-b167-4397-bb13-a9ef77e413f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-d8sgs"
Oct 30 23:07:44.464712 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:44.464631    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d8945b9-b167-4397-bb13-a9ef77e413f7-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-d8sgs\" (UID: \"4d8945b9-b167-4397-bb13-a9ef77e413f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-d8sgs"
Oct 30 23:07:44.464712 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:44.464679    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d8945b9-b167-4397-bb13-a9ef77e413f7-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-d8sgs\" (UID: \"4d8945b9-b167-4397-bb13-a9ef77e413f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-d8sgs"
Oct 30 23:07:44.464781 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:44.464766    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d8945b9-b167-4397-bb13-a9ef77e413f7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-d8sgs\" (UID: \"4d8945b9-b167-4397-bb13-a9ef77e413f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-d8sgs"
Oct 30 23:07:44.464818 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:44.464786    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d8945b9-b167-4397-bb13-a9ef77e413f7-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-d8sgs\" (UID: \"4d8945b9-b167-4397-bb13-a9ef77e413f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-d8sgs"
Oct 30 23:07:44.566100 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:44.566072    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d8945b9-b167-4397-bb13-a9ef77e413f7-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-d8sgs\" (UID: \"4d8945b9-b167-4397-bb13-a9ef77e413f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-d8sgs"
Oct 30 23:07:44.566239 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:44.566107    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d8945b9-b167-4397-bb13-a9ef77e413f7-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-d8sgs\" (UID: \"4d8945b9-b167-4397-bb13-a9ef77e413f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-d8sgs"
Oct 30 23:07:44.566239 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:44.566139    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d8945b9-b167-4397-bb13-a9ef77e413f7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-d8sgs\" (UID: \"4d8945b9-b167-4397-bb13-a9ef77e413f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-d8sgs"
Oct 30 23:07:44.566239 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:44.566166    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d8945b9-b167-4397-bb13-a9ef77e413f7-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-d8sgs\" (UID: \"4d8945b9-b167-4397-bb13-a9ef77e413f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-d8sgs"
Oct 30 23:07:44.566239 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:44.566232    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d8945b9-b167-4397-bb13-a9ef77e413f7-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-d8sgs\" (UID: \"4d8945b9-b167-4397-bb13-a9ef77e413f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-d8sgs"
Oct 30 23:07:44.566473 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:44.566276    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d8945b9-b167-4397-bb13-a9ef77e413f7-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-d8sgs\" (UID: \"4d8945b9-b167-4397-bb13-a9ef77e413f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-d8sgs"
Oct 30 23:07:44.566473 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:44.566308    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d8945b9-b167-4397-bb13-a9ef77e413f7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-d8sgs\" (UID: \"4d8945b9-b167-4397-bb13-a9ef77e413f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-d8sgs"
Oct 30 23:07:44.566473 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:44.566403    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d8945b9-b167-4397-bb13-a9ef77e413f7-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-d8sgs\" (UID: \"4d8945b9-b167-4397-bb13-a9ef77e413f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-d8sgs"
Oct 30 23:07:44.566634 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:44.566467    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkdhg\" (UniqueName: \"kubernetes.io/projected/4d8945b9-b167-4397-bb13-a9ef77e413f7-kube-api-access-jkdhg\") pod \"maas-default-gateway-openshift-default-6db586dd6c-d8sgs\" (UID: \"4d8945b9-b167-4397-bb13-a9ef77e413f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-d8sgs"
Oct 30 23:07:44.566634 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:44.566522    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d8945b9-b167-4397-bb13-a9ef77e413f7-workload-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-d8sgs\" (UID: \"4d8945b9-b167-4397-bb13-a9ef77e413f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-d8sgs"
Oct 30 23:07:44.566728 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:44.566645    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d8945b9-b167-4397-bb13-a9ef77e413f7-credential-socket\") pod \"maas-default-gateway-openshift-default-6db586dd6c-d8sgs\" (UID: \"4d8945b9-b167-4397-bb13-a9ef77e413f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-d8sgs"
Oct 30 23:07:44.566728 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:44.566716    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d8945b9-b167-4397-bb13-a9ef77e413f7-workload-certs\") pod \"maas-default-gateway-openshift-default-6db586dd6c-d8sgs\" (UID: \"4d8945b9-b167-4397-bb13-a9ef77e413f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-d8sgs"
Oct 30 23:07:44.566844 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:44.566824    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d8945b9-b167-4397-bb13-a9ef77e413f7-istio-data\") pod \"maas-default-gateway-openshift-default-6db586dd6c-d8sgs\" (UID: \"4d8945b9-b167-4397-bb13-a9ef77e413f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-d8sgs"
Oct 30 23:07:44.567063 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:44.567023    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d8945b9-b167-4397-bb13-a9ef77e413f7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6db586dd6c-d8sgs\" (UID: \"4d8945b9-b167-4397-bb13-a9ef77e413f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-d8sgs"
Oct 30 23:07:44.568785 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:44.568756    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d8945b9-b167-4397-bb13-a9ef77e413f7-istio-envoy\") pod \"maas-default-gateway-openshift-default-6db586dd6c-d8sgs\" (UID: \"4d8945b9-b167-4397-bb13-a9ef77e413f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-d8sgs"
Oct 30 23:07:44.568940 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:44.568915    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d8945b9-b167-4397-bb13-a9ef77e413f7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6db586dd6c-d8sgs\" (UID: \"4d8945b9-b167-4397-bb13-a9ef77e413f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-d8sgs"
Oct 30 23:07:44.602016 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:44.601990    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d8945b9-b167-4397-bb13-a9ef77e413f7-istio-token\") pod \"maas-default-gateway-openshift-default-6db586dd6c-d8sgs\" (UID: \"4d8945b9-b167-4397-bb13-a9ef77e413f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-d8sgs"
Oct 30 23:07:44.602171 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:44.602152    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkdhg\" (UniqueName: \"kubernetes.io/projected/4d8945b9-b167-4397-bb13-a9ef77e413f7-kube-api-access-jkdhg\") pod \"maas-default-gateway-openshift-default-6db586dd6c-d8sgs\" (UID: \"4d8945b9-b167-4397-bb13-a9ef77e413f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-d8sgs"
Oct 30 23:07:44.697196 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:44.697140    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-d8sgs"
Oct 30 23:07:45.013336 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:45.013204    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-d8sgs"]
Oct 30 23:07:45.017317 ip-10-0-130-77 kubenswrapper[4477]: W1030 23:07:45.017289    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d8945b9_b167_4397_bb13_a9ef77e413f7.slice/crio-6c757737c4b9bf3b99b38f8f25d1070600c03618b3dfd63729b04e9be6fe0498 WatchSource:0}: Error finding container 6c757737c4b9bf3b99b38f8f25d1070600c03618b3dfd63729b04e9be6fe0498: Status 404 returned error can't find the container with id 6c757737c4b9bf3b99b38f8f25d1070600c03618b3dfd63729b04e9be6fe0498
Oct 30 23:07:45.019638 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:45.019604    4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 23:07:45.019713 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:45.019671    4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 23:07:45.019713 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:45.019700    4477 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Oct 30 23:07:45.520854 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:45.520809    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-d8sgs" event={"ID":"4d8945b9-b167-4397-bb13-a9ef77e413f7","Type":"ContainerStarted","Data":"4427fc71a2b370e7ed9429ec2b6542037a245745d68ea3ac4c85174cbd414185"}
Oct 30 23:07:45.521330 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:45.520869    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-d8sgs" event={"ID":"4d8945b9-b167-4397-bb13-a9ef77e413f7","Type":"ContainerStarted","Data":"6c757737c4b9bf3b99b38f8f25d1070600c03618b3dfd63729b04e9be6fe0498"}
Oct 30 23:07:45.592062 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:45.591529    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-d8sgs" podStartSLOduration=1.5915107019999999 podStartE2EDuration="1.591510702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:07:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:07:45.574222205 +0000 UTC m=+3649.500148441" watchObservedRunningTime="2026-10-30 23:07:45.591510702 +0000 UTC m=+3649.517436905"
Oct 30 23:07:45.697490 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:45.697454    4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-d8sgs"
Oct 30 23:07:46.699461 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:46.699428    4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-d8sgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body=
Oct 30 23:07:46.699842 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:46.699478    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-d8sgs" podUID="4d8945b9-b167-4397-bb13-a9ef77e413f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused"
Oct 30 23:07:47.639088 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:47.639052    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6bz92"]
Oct 30 23:07:47.697897 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:47.697863    4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-d8sgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body=
Oct 30 23:07:47.698101 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:47.697916    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-d8sgs" podUID="4d8945b9-b167-4397-bb13-a9ef77e413f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused"
Oct 30 23:07:48.698598 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:48.698559    4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-d8sgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body=
Oct 30 23:07:48.699159 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:48.699126    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-d8sgs" podUID="4d8945b9-b167-4397-bb13-a9ef77e413f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused"
Oct 30 23:07:49.700433 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:49.700382    4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-d8sgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body=
Oct 30 23:07:49.700975 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:49.700451    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-d8sgs" podUID="4d8945b9-b167-4397-bb13-a9ef77e413f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused"
Oct 30 23:07:50.698297 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:50.698257    4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-d8sgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body=
Oct 30 23:07:50.698458 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:50.698311    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-d8sgs" podUID="4d8945b9-b167-4397-bb13-a9ef77e413f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused"
Oct 30 23:07:51.697871 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:51.697834    4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-d8sgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused" start-of-body=
Oct 30 23:07:51.698322 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:51.697914    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-d8sgs" podUID="4d8945b9-b167-4397-bb13-a9ef77e413f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": dial tcp 10.134.0.150:15021: connect: connection refused"
Oct 30 23:07:53.699443 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:53.699411    4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-d8sgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 23:07:53.699848 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:53.699463    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-d8sgs" podUID="4d8945b9-b167-4397-bb13-a9ef77e413f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:07:54.741220 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:54.741176    4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-d8sgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 23:07:54.741704 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:54.741233    4477 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-d8sgs" podUID="4d8945b9-b167-4397-bb13-a9ef77e413f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:07:55.192162 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:55.192135    4477 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-d8sgs"
Oct 30 23:07:55.192539 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:55.192519    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-d8sgs"
Oct 30 23:07:55.193250 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:07:55.193224    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-d8sgs"
Oct 30 23:07:56.265271 ip-10-0-130-77 kubenswrapper[4477]: E1030 23:07:56.265230    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 23:08:05.620925 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:08:05.620892    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6bz92"]
Oct 30 23:08:21.398287 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:08:21.398244    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6bz92"]
Oct 30 23:08:26.265452 ip-10-0-130-77 kubenswrapper[4477]: E1030 23:08:26.265412    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 23:08:44.712507 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:08:44.712464    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6bz92"]
Oct 30 23:08:55.083530 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:08:55.083478    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6bz92"]
Oct 30 23:08:56.266204 ip-10-0-130-77 kubenswrapper[4477]: E1030 23:08:56.266163    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 23:08:59.811706 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:08:59.811672    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6bz92"]
Oct 30 23:09:16.590879 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:09:16.590780    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6bz92"]
Oct 30 23:09:22.237523 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:09:22.237462    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6bz92"]
Oct 30 23:09:26.266035 ip-10-0-130-77 kubenswrapper[4477]: E1030 23:09:26.265996    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 23:09:38.095569 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:09:38.095528    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6bz92"]
Oct 30 23:09:42.620727 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:09:42.620688    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6bz92"]
Oct 30 23:09:56.266183 ip-10-0-130-77 kubenswrapper[4477]: E1030 23:09:56.266143    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 23:09:58.279604 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:09:58.279574    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6bz92"]
Oct 30 23:10:03.764756 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:10:03.764718    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6bz92"]
Oct 30 23:10:18.386437 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:10:18.386401    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6bz92"]
Oct 30 23:10:26.266100 ip-10-0-130-77 kubenswrapper[4477]: E1030 23:10:26.266064    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 23:10:29.035606 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:10:29.035566    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6bz92"]
Oct 30 23:10:35.795250 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:10:35.795210    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6bz92"]
Oct 30 23:10:56.265463 ip-10-0-130-77 kubenswrapper[4477]: E1030 23:10:56.265382    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 23:11:02.103206 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:11:02.103172    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6bz92"]
Oct 30 23:11:20.109397 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:11:20.109360    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6bz92"]
Oct 30 23:11:26.268710 ip-10-0-130-77 kubenswrapper[4477]: E1030 23:11:26.268663    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 23:11:30.511621 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:11:30.511582    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6bz92"]
Oct 30 23:11:56.268806 ip-10-0-130-77 kubenswrapper[4477]: E1030 23:11:56.268770    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 23:11:57.190917 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:11:57.190802    4477 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jcf8l_015f218d-bf1f-4e7e-bb5d-b805af003246/ovn-acl-logging/0.log"
Oct 30 23:11:57.190917 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:11:57.190847    4477 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jcf8l_015f218d-bf1f-4e7e-bb5d-b805af003246/ovn-acl-logging/0.log"
Oct 30 23:12:03.188845 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:12:03.188810    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6bz92"]
Oct 30 23:12:17.726430 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:12:17.726329    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6bz92"]
Oct 30 23:12:26.266663 ip-10-0-130-77 kubenswrapper[4477]: E1030 23:12:26.266628    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 23:12:33.520754 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:12:33.520695    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6bz92"]
Oct 30 23:12:56.264301 ip-10-0-130-77 kubenswrapper[4477]: E1030 23:12:56.264266    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 23:12:59.705639 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:12:59.705607    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6bz92"]
Oct 30 23:13:13.438686 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:13:13.438643    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6bz92"]
Oct 30 23:13:26.265207 ip-10-0-130-77 kubenswrapper[4477]: E1030 23:13:26.265165    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 23:13:56.265218 ip-10-0-130-77 kubenswrapper[4477]: E1030 23:13:56.265134    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 23:14:26.264954 ip-10-0-130-77 kubenswrapper[4477]: E1030 23:14:26.264917    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 23:14:56.268225 ip-10-0-130-77 kubenswrapper[4477]: E1030 23:14:56.268177    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 23:15:00.512794 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:15:00.512752    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6bz92"]
Oct 30 23:15:05.001790 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:15:05.001757    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-dbwzj"]
Oct 30 23:15:05.005681 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:15:05.005663    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-dbwzj"
Oct 30 23:15:05.019747 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:15:05.019727    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:15:05.030772 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:15:05.030753    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-vp5mn\""
Oct 30 23:15:05.032711 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:15:05.032688    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-dbwzj"]
Oct 30 23:15:05.057093 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:15:05.057060    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e72db958-5c90-436d-8d61-17fe172ef7a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-dbwzj\" (UID: \"e72db958-5c90-436d-8d61-17fe172ef7a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-dbwzj"
Oct 30 23:15:05.057220 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:15:05.057107    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e72db958-5c90-436d-8d61-17fe172ef7a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-dbwzj\" (UID: \"e72db958-5c90-436d-8d61-17fe172ef7a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-dbwzj"
Oct 30 23:15:05.057220 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:15:05.057129    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72bg2\" (UniqueName: \"kubernetes.io/projected/e72db958-5c90-436d-8d61-17fe172ef7a1-kube-api-access-72bg2\") pod \"maas-api-revoke-sub-d576e68b-dbwzj\" (UID: \"e72db958-5c90-436d-8d61-17fe172ef7a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-dbwzj"
Oct 30 23:15:05.158390 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:15:05.158365    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e72db958-5c90-436d-8d61-17fe172ef7a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-dbwzj\" (UID: \"e72db958-5c90-436d-8d61-17fe172ef7a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-dbwzj"
Oct 30 23:15:05.158495 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:15:05.158397    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e72db958-5c90-436d-8d61-17fe172ef7a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-dbwzj\" (UID: \"e72db958-5c90-436d-8d61-17fe172ef7a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-dbwzj"
Oct 30 23:15:05.158495 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:15:05.158416    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72bg2\" (UniqueName: \"kubernetes.io/projected/e72db958-5c90-436d-8d61-17fe172ef7a1-kube-api-access-72bg2\") pod \"maas-api-revoke-sub-d576e68b-dbwzj\" (UID: \"e72db958-5c90-436d-8d61-17fe172ef7a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-dbwzj"
Oct 30 23:15:05.159019 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:15:05.158998    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e72db958-5c90-436d-8d61-17fe172ef7a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-dbwzj\" (UID: \"e72db958-5c90-436d-8d61-17fe172ef7a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-dbwzj"
Oct 30 23:15:05.160793 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:15:05.160771    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e72db958-5c90-436d-8d61-17fe172ef7a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-dbwzj\" (UID: \"e72db958-5c90-436d-8d61-17fe172ef7a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-dbwzj"
Oct 30 23:15:05.177122 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:15:05.177103    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72bg2\" (UniqueName: \"kubernetes.io/projected/e72db958-5c90-436d-8d61-17fe172ef7a1-kube-api-access-72bg2\") pod \"maas-api-revoke-sub-d576e68b-dbwzj\" (UID: \"e72db958-5c90-436d-8d61-17fe172ef7a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-dbwzj"
Oct 30 23:15:05.318255 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:15:05.318201    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-dbwzj"
Oct 30 23:15:05.608480 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:15:05.608395    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-dbwzj"]
Oct 30 23:15:05.613797 ip-10-0-130-77 kubenswrapper[4477]: W1030 23:15:05.613770    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode72db958_5c90_436d_8d61_17fe172ef7a1.slice/crio-64548ba2fe1e6419677467e463f2b8b0e53d86d78c3604a62460d7fa4895a2a0 WatchSource:0}: Error finding container 64548ba2fe1e6419677467e463f2b8b0e53d86d78c3604a62460d7fa4895a2a0: Status 404 returned error can't find the container with id 64548ba2fe1e6419677467e463f2b8b0e53d86d78c3604a62460d7fa4895a2a0
Oct 30 23:15:05.652876 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:15:05.652852    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-dbwzj" event={"ID":"e72db958-5c90-436d-8d61-17fe172ef7a1","Type":"ContainerStarted","Data":"64548ba2fe1e6419677467e463f2b8b0e53d86d78c3604a62460d7fa4895a2a0"}
Oct 30 23:15:06.659339 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:15:06.659311    4477 generic.go:356] "Generic (PLEG): container finished" podID="e72db958-5c90-436d-8d61-17fe172ef7a1" containerID="243e7d2a4bf5793356fbf78203420fe58cf18fec40db8e5bdb7fdf80d9e9a0fa" exitCode=0
Oct 30 23:15:06.659697 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:15:06.659351    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-dbwzj" event={"ID":"e72db958-5c90-436d-8d61-17fe172ef7a1","Type":"ContainerDied","Data":"243e7d2a4bf5793356fbf78203420fe58cf18fec40db8e5bdb7fdf80d9e9a0fa"}
Oct 30 23:15:07.800456 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:15:07.800428    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-dbwzj"
Oct 30 23:15:07.881987 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:15:07.881959    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e72db958-5c90-436d-8d61-17fe172ef7a1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e72db958-5c90-436d-8d61-17fe172ef7a1-maas-api-xks-ca\") pod \"e72db958-5c90-436d-8d61-17fe172ef7a1\" (UID: \"e72db958-5c90-436d-8d61-17fe172ef7a1\") "
Oct 30 23:15:07.882144 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:15:07.882112    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e72db958-5c90-436d-8d61-17fe172ef7a1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e72db958-5c90-436d-8d61-17fe172ef7a1-maas-api-openshift-ca\") pod \"e72db958-5c90-436d-8d61-17fe172ef7a1\" (UID: \"e72db958-5c90-436d-8d61-17fe172ef7a1\") "
Oct 30 23:15:07.882193 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:15:07.882151    4477 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e72db958-5c90-436d-8d61-17fe172ef7a1-kube-api-access-72bg2\" (UniqueName: \"kubernetes.io/projected/e72db958-5c90-436d-8d61-17fe172ef7a1-kube-api-access-72bg2\") pod \"e72db958-5c90-436d-8d61-17fe172ef7a1\" (UID: \"e72db958-5c90-436d-8d61-17fe172ef7a1\") "
Oct 30 23:15:07.882495 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:15:07.882466    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e72db958-5c90-436d-8d61-17fe172ef7a1-maas-api-openshift-ca" pod "e72db958-5c90-436d-8d61-17fe172ef7a1" (UID: "e72db958-5c90-436d-8d61-17fe172ef7a1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:15:07.883983 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:15:07.883957    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e72db958-5c90-436d-8d61-17fe172ef7a1-maas-api-xks-ca" pod "e72db958-5c90-436d-8d61-17fe172ef7a1" (UID: "e72db958-5c90-436d-8d61-17fe172ef7a1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:15:07.884367 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:15:07.884345    4477 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e72db958-5c90-436d-8d61-17fe172ef7a1-kube-api-access-72bg2" pod "e72db958-5c90-436d-8d61-17fe172ef7a1" (UID: "e72db958-5c90-436d-8d61-17fe172ef7a1"). InnerVolumeSpecName "kube-api-access-72bg2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:15:07.983082 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:15:07.983018    4477 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e72db958-5c90-436d-8d61-17fe172ef7a1-maas-api-xks-ca\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:15:07.983082 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:15:07.983049    4477 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e72db958-5c90-436d-8d61-17fe172ef7a1-maas-api-openshift-ca\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:15:07.983082 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:15:07.983059    4477 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72bg2\" (UniqueName: \"kubernetes.io/projected/e72db958-5c90-436d-8d61-17fe172ef7a1-kube-api-access-72bg2\") on node \"ip-10-0-130-77.ec2.internal\" DevicePath \"\""
Oct 30 23:15:08.669275 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:15:08.669247    4477 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-dbwzj"
Oct 30 23:15:08.669446 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:15:08.669247    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-dbwzj" event={"ID":"e72db958-5c90-436d-8d61-17fe172ef7a1","Type":"ContainerDied","Data":"64548ba2fe1e6419677467e463f2b8b0e53d86d78c3604a62460d7fa4895a2a0"}
Oct 30 23:15:08.669446 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:15:08.669364    4477 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="64548ba2fe1e6419677467e463f2b8b0e53d86d78c3604a62460d7fa4895a2a0"
Oct 30 23:15:16.215713 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:15:16.215679    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6bz92"]
Oct 30 23:15:22.507395 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:15:22.507354    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6bz92"]
Oct 30 23:15:26.265651 ip-10-0-130-77 kubenswrapper[4477]: E1030 23:15:26.265609    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 23:15:37.522838 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:15:37.522804    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6bz92"]
Oct 30 23:15:47.222891 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:15:47.222853    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6bz92"]
Oct 30 23:15:56.266817 ip-10-0-130-77 kubenswrapper[4477]: E1030 23:15:56.266777    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 23:16:00.729425 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:16:00.729386    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6bz92"]
Oct 30 23:16:18.721647 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:16:18.721609    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6bz92"]
Oct 30 23:16:26.266816 ip-10-0-130-77 kubenswrapper[4477]: E1030 23:16:26.266773    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 23:16:43.344036 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:16:43.343942    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6bz92"]
Oct 30 23:16:56.268741 ip-10-0-130-77 kubenswrapper[4477]: E1030 23:16:56.268708    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 23:16:57.243218 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:16:57.243101    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6bz92"]
Oct 30 23:16:57.245088 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:16:57.245065    4477 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jcf8l_015f218d-bf1f-4e7e-bb5d-b805af003246/ovn-acl-logging/0.log"
Oct 30 23:16:57.248546 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:16:57.248525    4477 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jcf8l_015f218d-bf1f-4e7e-bb5d-b805af003246/ovn-acl-logging/0.log"
Oct 30 23:17:12.010010 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:17:12.009976    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6bz92"]
Oct 30 23:17:26.265908 ip-10-0-130-77 kubenswrapper[4477]: E1030 23:17:26.265872    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 23:17:31.428545 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:17:31.428510    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6bz92"]
Oct 30 23:17:47.125946 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:17:47.125913    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6bz92"]
Oct 30 23:17:56.265260 ip-10-0-130-77 kubenswrapper[4477]: E1030 23:17:56.265216    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 23:17:59.093288 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:17:59.093249    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6bz92"]
Oct 30 23:18:08.015352 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:18:08.015315    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6bz92"]
Oct 30 23:18:16.307224 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:18:16.307145    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6bz92"]
Oct 30 23:18:23.819186 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:18:23.819150    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6bz92"]
Oct 30 23:18:26.265364 ip-10-0-130-77 kubenswrapper[4477]: E1030 23:18:26.265325    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 23:18:39.901427 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:18:39.901387    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6bz92"]
Oct 30 23:18:48.359967 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:18:48.359934    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-567764f58f-zlvqn"]
Oct 30 23:18:48.360681 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:18:48.360579    4477 cpu_manager.go:414] "containerMap: removing container" podUID="e72db958-5c90-436d-8d61-17fe172ef7a1" containerName="revoke-keys"
Oct 30 23:18:48.360681 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:18:48.360604    4477 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e72db958-5c90-436d-8d61-17fe172ef7a1" containerName="revoke-keys"
Oct 30 23:18:48.360793 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:18:48.360778    4477 memory_manager.go:365] "RemoveStaleState removing state" podUID="e72db958-5c90-436d-8d61-17fe172ef7a1" containerName="revoke-keys"
Oct 30 23:18:48.363941 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:18:48.363923    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-567764f58f-zlvqn"
Oct 30 23:18:48.389298 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:18:48.389224    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 23:18:48.389439 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:18:48.389322    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-l95g7\""
Oct 30 23:18:48.389439 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:18:48.389355    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Oct 30 23:18:48.391836 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:18:48.391812    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-567764f58f-zlvqn"]
Oct 30 23:18:48.448826 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:18:48.448793    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cz64g\" (UniqueName: \"kubernetes.io/projected/48743a0a-4675-404e-addb-ec01270a7ea1-kube-api-access-cz64g\") pod \"maas-api-567764f58f-zlvqn\" (UID: \"48743a0a-4675-404e-addb-ec01270a7ea1\") " pod="odh-ai-gateway-infra/maas-api-567764f58f-zlvqn"
Oct 30 23:18:48.449005 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:18:48.448845    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48743a0a-4675-404e-addb-ec01270a7ea1-metrics-certs\") pod \"maas-api-567764f58f-zlvqn\" (UID: \"48743a0a-4675-404e-addb-ec01270a7ea1\") " pod="odh-ai-gateway-infra/maas-api-567764f58f-zlvqn"
Oct 30 23:18:48.449005 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:18:48.448955    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48743a0a-4675-404e-addb-ec01270a7ea1-openshift-service-ca\") pod \"maas-api-567764f58f-zlvqn\" (UID: \"48743a0a-4675-404e-addb-ec01270a7ea1\") " pod="odh-ai-gateway-infra/maas-api-567764f58f-zlvqn"
Oct 30 23:18:48.449161 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:18:48.449013    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48743a0a-4675-404e-addb-ec01270a7ea1-maas-api-tls\") pod \"maas-api-567764f58f-zlvqn\" (UID: \"48743a0a-4675-404e-addb-ec01270a7ea1\") " pod="odh-ai-gateway-infra/maas-api-567764f58f-zlvqn"
Oct 30 23:18:48.550197 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:18:48.550167    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cz64g\" (UniqueName: \"kubernetes.io/projected/48743a0a-4675-404e-addb-ec01270a7ea1-kube-api-access-cz64g\") pod \"maas-api-567764f58f-zlvqn\" (UID: \"48743a0a-4675-404e-addb-ec01270a7ea1\") " pod="odh-ai-gateway-infra/maas-api-567764f58f-zlvqn"
Oct 30 23:18:48.550390 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:18:48.550207    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48743a0a-4675-404e-addb-ec01270a7ea1-metrics-certs\") pod \"maas-api-567764f58f-zlvqn\" (UID: \"48743a0a-4675-404e-addb-ec01270a7ea1\") " pod="odh-ai-gateway-infra/maas-api-567764f58f-zlvqn"
Oct 30 23:18:48.550390 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:18:48.550229    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48743a0a-4675-404e-addb-ec01270a7ea1-openshift-service-ca\") pod \"maas-api-567764f58f-zlvqn\" (UID: \"48743a0a-4675-404e-addb-ec01270a7ea1\") " pod="odh-ai-gateway-infra/maas-api-567764f58f-zlvqn"
Oct 30 23:18:48.550390 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:18:48.550282    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48743a0a-4675-404e-addb-ec01270a7ea1-maas-api-tls\") pod \"maas-api-567764f58f-zlvqn\" (UID: \"48743a0a-4675-404e-addb-ec01270a7ea1\") " pod="odh-ai-gateway-infra/maas-api-567764f58f-zlvqn"
Oct 30 23:18:48.551353 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:18:48.551311    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/48743a0a-4675-404e-addb-ec01270a7ea1-openshift-service-ca\") pod \"maas-api-567764f58f-zlvqn\" (UID: \"48743a0a-4675-404e-addb-ec01270a7ea1\") " pod="odh-ai-gateway-infra/maas-api-567764f58f-zlvqn"
Oct 30 23:18:48.552971 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:18:48.552946    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/48743a0a-4675-404e-addb-ec01270a7ea1-maas-api-tls\") pod \"maas-api-567764f58f-zlvqn\" (UID: \"48743a0a-4675-404e-addb-ec01270a7ea1\") " pod="odh-ai-gateway-infra/maas-api-567764f58f-zlvqn"
Oct 30 23:18:48.556327 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:18:48.556302    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48743a0a-4675-404e-addb-ec01270a7ea1-metrics-certs\") pod \"maas-api-567764f58f-zlvqn\" (UID: \"48743a0a-4675-404e-addb-ec01270a7ea1\") " pod="odh-ai-gateway-infra/maas-api-567764f58f-zlvqn"
Oct 30 23:18:48.578689 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:18:48.578658    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cz64g\" (UniqueName: \"kubernetes.io/projected/48743a0a-4675-404e-addb-ec01270a7ea1-kube-api-access-cz64g\") pod \"maas-api-567764f58f-zlvqn\" (UID: \"48743a0a-4675-404e-addb-ec01270a7ea1\") " pod="odh-ai-gateway-infra/maas-api-567764f58f-zlvqn"
Oct 30 23:18:48.677474 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:18:48.677446    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-567764f58f-zlvqn"
Oct 30 23:18:48.973353 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:18:48.973219    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-567764f58f-zlvqn"]
Oct 30 23:18:48.978151 ip-10-0-130-77 kubenswrapper[4477]: W1030 23:18:48.978121    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48743a0a_4675_404e_addb_ec01270a7ea1.slice/crio-cd777c211b128f70d82cba8d638235b1d6e544ec62210a4e642cbf6432018a08 WatchSource:0}: Error finding container cd777c211b128f70d82cba8d638235b1d6e544ec62210a4e642cbf6432018a08: Status 404 returned error can't find the container with id cd777c211b128f70d82cba8d638235b1d6e544ec62210a4e642cbf6432018a08
Oct 30 23:18:48.979515 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:18:48.979497    4477 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 23:18:49.791445 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:18:49.791408    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-567764f58f-zlvqn" event={"ID":"48743a0a-4675-404e-addb-ec01270a7ea1","Type":"ContainerStarted","Data":"c15659377ca4745bf1d1293f5fb3b093126d359f57ffd2f50f864b6afeefc64e"}
Oct 30 23:18:49.791841 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:18:49.791464    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-567764f58f-zlvqn" event={"ID":"48743a0a-4675-404e-addb-ec01270a7ea1","Type":"ContainerStarted","Data":"cd777c211b128f70d82cba8d638235b1d6e544ec62210a4e642cbf6432018a08"}
Oct 30 23:18:49.791841 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:18:49.791492    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-567764f58f-zlvqn"
Oct 30 23:18:49.830101 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:18:49.830029    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-567764f58f-zlvqn" podStartSLOduration=1.482014033 podStartE2EDuration="1.830015639s" totalImagesPullingTime="348.001606ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:18:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:18:49.829565123 +0000 UTC m=+4313.755491329" watchObservedRunningTime="2026-10-30 23:18:49.830015639 +0000 UTC m=+4313.755941845"
Oct 30 23:18:55.803428 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:18:55.803398    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-567764f58f-zlvqn"
Oct 30 23:18:56.265236 ip-10-0-130-77 kubenswrapper[4477]: E1030 23:18:56.265199    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 23:18:57.389690 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:18:57.389655    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6bz92"]
Oct 30 23:19:20.033828 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:19:20.033782    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6bz92"]
Oct 30 23:19:26.266012 ip-10-0-130-77 kubenswrapper[4477]: E1030 23:19:26.265960    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 23:19:33.848499 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:19:33.848453    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6bz92"]
Oct 30 23:19:56.265198 ip-10-0-130-77 kubenswrapper[4477]: E1030 23:19:56.265119    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 23:20:08.043729 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:20:08.043695    4477 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-dbwzj"]
Oct 30 23:20:08.055153 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:20:08.055123    4477 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-dbwzj"]
Oct 30 23:20:08.191581 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:20:08.191551    4477 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e72db958-5c90-436d-8d61-17fe172ef7a1" path="/var/lib/kubelet/pods/e72db958-5c90-436d-8d61-17fe172ef7a1/volumes"
Oct 30 23:20:26.264864 ip-10-0-130-77 kubenswrapper[4477]: E1030 23:20:26.264824    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 23:20:56.266463 ip-10-0-130-77 kubenswrapper[4477]: E1030 23:20:56.266420    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 23:21:22.512854 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:21:22.512770    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6bz92"]
Oct 30 23:21:26.264973 ip-10-0-130-77 kubenswrapper[4477]: E1030 23:21:26.264931    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 23:21:30.511007 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:21:30.510972    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6bz92"]
Oct 30 23:21:38.719657 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:21:38.719623    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6bz92"]
Oct 30 23:21:41.233204 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:21:41.233160    4477 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6db586dd6c-d8sgs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.150:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 23:21:41.233735 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:21:41.233237    4477 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6db586dd6c-d8sgs" podUID="4d8945b9-b167-4397-bb13-a9ef77e413f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.150:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:21:56.267678 ip-10-0-130-77 kubenswrapper[4477]: E1030 23:21:56.267638    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 23:21:56.619378 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:21:56.619296    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6bz92"]
Oct 30 23:21:57.279959 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:21:57.279852    4477 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jcf8l_015f218d-bf1f-4e7e-bb5d-b805af003246/ovn-acl-logging/0.log"
Oct 30 23:21:57.285862 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:21:57.285843    4477 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jcf8l_015f218d-bf1f-4e7e-bb5d-b805af003246/ovn-acl-logging/0.log"
Oct 30 23:21:57.748972 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:21:57.748941    4477 scope.go:122] "RemoveContainer" containerID="243e7d2a4bf5793356fbf78203420fe58cf18fec40db8e5bdb7fdf80d9e9a0fa"
Oct 30 23:22:25.003466 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:22:25.003432    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-6bz92"]
Oct 30 23:22:26.265582 ip-10-0-130-77 kubenswrapper[4477]: E1030 23:22:26.265548    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 23:22:35.852811 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:22:35.852783    4477 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-ddzfb_95aafcdd-d16c-40af-8dd3-e0fcefcbf5f9/manager/0.log"
Oct 30 23:22:36.283576 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:22:36.283546    4477 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-8lgfv_8bb0b4e6-3fa7-4257-b686-c6d84294038e/manager/0.log"
Oct 30 23:22:36.754011 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:22:36.753976    4477 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-f29pm_1d33baf4-42cd-409c-a3fe-8a0bfcf7b1c5/server/0.log"
Oct 30 23:22:36.949074 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:22:36.949028    4477 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-ssczl_12bab827-4026-4cfb-8182-cb686e075e8c/manager/0.log"
Oct 30 23:22:37.172537 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:22:37.172502    4477 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-x69f9_2d3c4dc5-460f-4d28-bad8-e97c462fe021/proxy/0.log"
Oct 30 23:22:38.115695 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:22:38.115661    4477 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-567764f58f-zlvqn_48743a0a-4675-404e-addb-ec01270a7ea1/maas-api/0.log"
Oct 30 23:22:39.462920 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:22:39.462823    4477 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-lx5nm_8a241719-733b-4527-85a9-727977fc863c/postgres/0.log"
Oct 30 23:22:41.364037 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:22:41.364002    4477 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-frfbm_41c589fb-1f6a-442e-ad41-cd6a7ba1ae39/manager/0.log"
Oct 30 23:22:41.549489 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:22:41.549463    4477 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-mfvqt_b398c8cf-bcf4-4e3a-98d2-bbf862919745/kuadrant-console-plugin/0.log"
Oct 30 23:22:41.963015 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:22:41.962982    4477 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-6bz92_b0d95e35-ac33-4203-a312-e7b072d8445a/limitador/0.log"
Oct 30 23:22:43.118504 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:22:43.118471    4477 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-wx4cl_6f02c360-27f4-4a25-8155-436e260be559/discovery/0.log"
Oct 30 23:22:43.531240 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:22:43.531210    4477 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6f865f994d-vg2nd_5fc4bd88-0db5-4608-ab12-88364e90d0c9/kube-auth-proxy/0.log"
Oct 30 23:22:44.167054 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:22:44.167020    4477 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-bjfxc_eeb7af55-f99f-411e-9f71-8ac1638c67ed/istio-proxy/0.log"
Oct 30 23:22:44.361622 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:22:44.361592    4477 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-d8sgs_4d8945b9-b167-4397-bb13-a9ef77e413f7/istio-proxy/0.log"
Oct 30 23:22:56.266881 ip-10-0-130-77 kubenswrapper[4477]: E1030 23:22:56.266841    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 23:22:58.364078 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:22:58.364000    4477 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-ddzfb_95aafcdd-d16c-40af-8dd3-e0fcefcbf5f9/manager/0.log"
Oct 30 23:22:58.775444 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:22:58.775400    4477 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-8lgfv_8bb0b4e6-3fa7-4257-b686-c6d84294038e/manager/0.log"
Oct 30 23:22:59.190621 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:22:59.190595    4477 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-f29pm_1d33baf4-42cd-409c-a3fe-8a0bfcf7b1c5/server/0.log"
Oct 30 23:22:59.389927 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:22:59.389892    4477 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-ssczl_12bab827-4026-4cfb-8182-cb686e075e8c/manager/0.log"
Oct 30 23:22:59.623940 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:22:59.623910    4477 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-x69f9_2d3c4dc5-460f-4d28-bad8-e97c462fe021/proxy/0.log"
Oct 30 23:23:00.316135 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:23:00.316106    4477 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-567764f58f-zlvqn_48743a0a-4675-404e-addb-ec01270a7ea1/maas-api/0.log"
Oct 30 23:23:01.450970 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:23:01.450929    4477 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-lx5nm_8a241719-733b-4527-85a9-727977fc863c/postgres/0.log"
Oct 30 23:23:03.088644 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:23:03.088585    4477 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-frfbm_41c589fb-1f6a-442e-ad41-cd6a7ba1ae39/manager/0.log"
Oct 30 23:23:03.278032 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:23:03.278007    4477 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-mfvqt_b398c8cf-bcf4-4e3a-98d2-bbf862919745/kuadrant-console-plugin/0.log"
Oct 30 23:23:03.711104 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:23:03.711014    4477 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-6bz92_b0d95e35-ac33-4203-a312-e7b072d8445a/limitador/0.log"
Oct 30 23:23:04.666332 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:23:04.666296    4477 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-wx4cl_6f02c360-27f4-4a25-8155-436e260be559/discovery/0.log"
Oct 30 23:23:05.041785 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:23:05.041693    4477 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6f865f994d-vg2nd_5fc4bd88-0db5-4608-ab12-88364e90d0c9/kube-auth-proxy/0.log"
Oct 30 23:23:05.631374 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:23:05.631343    4477 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-bjfxc_eeb7af55-f99f-411e-9f71-8ac1638c67ed/istio-proxy/0.log"
Oct 30 23:23:05.830656 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:23:05.830631    4477 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-d8sgs_4d8945b9-b167-4397-bb13-a9ef77e413f7/istio-proxy/0.log"
Oct 30 23:23:26.264851 ip-10-0-130-77 kubenswrapper[4477]: E1030 23:23:26.264814    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 23:23:56.264986 ip-10-0-130-77 kubenswrapper[4477]: E1030 23:23:56.264949    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 23:24:26.265891 ip-10-0-130-77 kubenswrapper[4477]: E1030 23:24:26.265852    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 23:24:32.936956 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:24:32.936924    4477 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-rjtc8_b8503b30-31d6-4454-a3fd-8bea4a527b04/global-pull-secret-syncer/0.log"
Oct 30 23:24:33.700434 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:24:33.700403    4477 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-mm655_a3d96078-3ec8-40aa-855a-d1ba4acf2e74/konnectivity-agent/0.log"
Oct 30 23:24:33.986145 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:24:33.986073    4477 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-77.ec2.internal_1929b84fce10dee760acfa96faf62b49/haproxy/0.log"
Oct 30 23:24:40.557246 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:24:40.557210    4477 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-frfbm_41c589fb-1f6a-442e-ad41-cd6a7ba1ae39/manager/0.log"
Oct 30 23:24:40.677766 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:24:40.677740    4477 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-mfvqt_b398c8cf-bcf4-4e3a-98d2-bbf862919745/kuadrant-console-plugin/0.log"
Oct 30 23:24:40.934294 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:24:40.934266    4477 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-6bz92_b0d95e35-ac33-4203-a312-e7b072d8445a/limitador/0.log"
Oct 30 23:24:42.059132 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:24:42.059103    4477 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-qktfv/perf-node-gather-daemonset-8sjs4"]
Oct 30 23:24:42.063103 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:24:42.063083    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qktfv/perf-node-gather-daemonset-8sjs4"
Oct 30 23:24:42.084187 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:24:42.084162    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qktfv/perf-node-gather-daemonset-8sjs4"]
Oct 30 23:24:42.093892 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:24:42.093870    4477 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-qktfv\"/\"default-dockercfg-qgg56\""
Oct 30 23:24:42.094075 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:24:42.093988    4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qktfv\"/\"openshift-service-ca.crt\""
Oct 30 23:24:42.094153 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:24:42.094086    4477 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qktfv\"/\"kube-root-ca.crt\""
Oct 30 23:24:42.145761 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:24:42.145733    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qf7n5\" (UniqueName: \"kubernetes.io/projected/3a94e4db-b581-4cf2-b961-4225304b0572-kube-api-access-qf7n5\") pod \"perf-node-gather-daemonset-8sjs4\" (UID: \"3a94e4db-b581-4cf2-b961-4225304b0572\") " pod="openshift-must-gather-qktfv/perf-node-gather-daemonset-8sjs4"
Oct 30 23:24:42.145874 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:24:42.145795    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3a94e4db-b581-4cf2-b961-4225304b0572-lib-modules\") pod \"perf-node-gather-daemonset-8sjs4\" (UID: \"3a94e4db-b581-4cf2-b961-4225304b0572\") " pod="openshift-must-gather-qktfv/perf-node-gather-daemonset-8sjs4"
Oct 30 23:24:42.145874 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:24:42.145833    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3a94e4db-b581-4cf2-b961-4225304b0572-sys\") pod \"perf-node-gather-daemonset-8sjs4\" (UID: \"3a94e4db-b581-4cf2-b961-4225304b0572\") " pod="openshift-must-gather-qktfv/perf-node-gather-daemonset-8sjs4"
Oct 30 23:24:42.146004 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:24:42.145943    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3a94e4db-b581-4cf2-b961-4225304b0572-proc\") pod \"perf-node-gather-daemonset-8sjs4\" (UID: \"3a94e4db-b581-4cf2-b961-4225304b0572\") " pod="openshift-must-gather-qktfv/perf-node-gather-daemonset-8sjs4"
Oct 30 23:24:42.146004 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:24:42.145982    4477 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3a94e4db-b581-4cf2-b961-4225304b0572-podres\") pod \"perf-node-gather-daemonset-8sjs4\" (UID: \"3a94e4db-b581-4cf2-b961-4225304b0572\") " pod="openshift-must-gather-qktfv/perf-node-gather-daemonset-8sjs4"
Oct 30 23:24:42.246835 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:24:42.246799    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3a94e4db-b581-4cf2-b961-4225304b0572-sys\") pod \"perf-node-gather-daemonset-8sjs4\" (UID: \"3a94e4db-b581-4cf2-b961-4225304b0572\") " pod="openshift-must-gather-qktfv/perf-node-gather-daemonset-8sjs4"
Oct 30 23:24:42.246835 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:24:42.246836    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3a94e4db-b581-4cf2-b961-4225304b0572-proc\") pod \"perf-node-gather-daemonset-8sjs4\" (UID: \"3a94e4db-b581-4cf2-b961-4225304b0572\") " pod="openshift-must-gather-qktfv/perf-node-gather-daemonset-8sjs4"
Oct 30 23:24:42.247025 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:24:42.246866    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3a94e4db-b581-4cf2-b961-4225304b0572-podres\") pod \"perf-node-gather-daemonset-8sjs4\" (UID: \"3a94e4db-b581-4cf2-b961-4225304b0572\") " pod="openshift-must-gather-qktfv/perf-node-gather-daemonset-8sjs4"
Oct 30 23:24:42.247025 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:24:42.246888    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3a94e4db-b581-4cf2-b961-4225304b0572-sys\") pod \"perf-node-gather-daemonset-8sjs4\" (UID: \"3a94e4db-b581-4cf2-b961-4225304b0572\") " pod="openshift-must-gather-qktfv/perf-node-gather-daemonset-8sjs4"
Oct 30 23:24:42.247025 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:24:42.246892    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qf7n5\" (UniqueName: \"kubernetes.io/projected/3a94e4db-b581-4cf2-b961-4225304b0572-kube-api-access-qf7n5\") pod \"perf-node-gather-daemonset-8sjs4\" (UID: \"3a94e4db-b581-4cf2-b961-4225304b0572\") " pod="openshift-must-gather-qktfv/perf-node-gather-daemonset-8sjs4"
Oct 30 23:24:42.247025 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:24:42.246969    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3a94e4db-b581-4cf2-b961-4225304b0572-proc\") pod \"perf-node-gather-daemonset-8sjs4\" (UID: \"3a94e4db-b581-4cf2-b961-4225304b0572\") " pod="openshift-must-gather-qktfv/perf-node-gather-daemonset-8sjs4"
Oct 30 23:24:42.247025 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:24:42.246978    4477 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3a94e4db-b581-4cf2-b961-4225304b0572-lib-modules\") pod \"perf-node-gather-daemonset-8sjs4\" (UID: \"3a94e4db-b581-4cf2-b961-4225304b0572\") " pod="openshift-must-gather-qktfv/perf-node-gather-daemonset-8sjs4"
Oct 30 23:24:42.247260 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:24:42.246969    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3a94e4db-b581-4cf2-b961-4225304b0572-podres\") pod \"perf-node-gather-daemonset-8sjs4\" (UID: \"3a94e4db-b581-4cf2-b961-4225304b0572\") " pod="openshift-must-gather-qktfv/perf-node-gather-daemonset-8sjs4"
Oct 30 23:24:42.247260 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:24:42.247066    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3a94e4db-b581-4cf2-b961-4225304b0572-lib-modules\") pod \"perf-node-gather-daemonset-8sjs4\" (UID: \"3a94e4db-b581-4cf2-b961-4225304b0572\") " pod="openshift-must-gather-qktfv/perf-node-gather-daemonset-8sjs4"
Oct 30 23:24:42.265207 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:24:42.265184    4477 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qf7n5\" (UniqueName: \"kubernetes.io/projected/3a94e4db-b581-4cf2-b961-4225304b0572-kube-api-access-qf7n5\") pod \"perf-node-gather-daemonset-8sjs4\" (UID: \"3a94e4db-b581-4cf2-b961-4225304b0572\") " pod="openshift-must-gather-qktfv/perf-node-gather-daemonset-8sjs4"
Oct 30 23:24:42.375863 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:24:42.375846    4477 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qktfv/perf-node-gather-daemonset-8sjs4"
Oct 30 23:24:42.667934 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:24:42.667840    4477 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qktfv/perf-node-gather-daemonset-8sjs4"]
Oct 30 23:24:42.672498 ip-10-0-130-77 kubenswrapper[4477]: W1030 23:24:42.672472    4477 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod3a94e4db_b581_4cf2_b961_4225304b0572.slice/crio-9c7950476fda9092cbd40b1fbac27d5e726958cafae4ae06746028fb615ebcdf WatchSource:0}: Error finding container 9c7950476fda9092cbd40b1fbac27d5e726958cafae4ae06746028fb615ebcdf: Status 404 returned error can't find the container with id 9c7950476fda9092cbd40b1fbac27d5e726958cafae4ae06746028fb615ebcdf
Oct 30 23:24:42.684440 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:24:42.684417    4477 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_4c435879-6dad-49c3-a9fa-fc1d7390d69e/alertmanager/0.log"
Oct 30 23:24:42.808146 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:24:42.808125    4477 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_4c435879-6dad-49c3-a9fa-fc1d7390d69e/config-reloader/0.log"
Oct 30 23:24:42.924980 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:24:42.924927    4477 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_4c435879-6dad-49c3-a9fa-fc1d7390d69e/kube-rbac-proxy-web/0.log"
Oct 30 23:24:43.041384 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:24:43.041366    4477 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_4c435879-6dad-49c3-a9fa-fc1d7390d69e/kube-rbac-proxy/0.log"
Oct 30 23:24:43.158640 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:24:43.158615    4477 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_4c435879-6dad-49c3-a9fa-fc1d7390d69e/kube-rbac-proxy-metric/0.log"
Oct 30 23:24:43.276384 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:24:43.276302    4477 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_4c435879-6dad-49c3-a9fa-fc1d7390d69e/prom-label-proxy/0.log"
Oct 30 23:24:43.394446 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:24:43.394422    4477 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_4c435879-6dad-49c3-a9fa-fc1d7390d69e/init-config-reloader/0.log"
Oct 30 23:24:43.497402 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:24:43.497368    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qktfv/perf-node-gather-daemonset-8sjs4" event={"ID":"3a94e4db-b581-4cf2-b961-4225304b0572","Type":"ContainerStarted","Data":"0c5fc4807f878b47b94f36cc0e459113d3c24dca32d0f7d9056729dd6562e780"}
Oct 30 23:24:43.497551 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:24:43.497530    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-qktfv/perf-node-gather-daemonset-8sjs4"
Oct 30 23:24:43.497624 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:24:43.497582    4477 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qktfv/perf-node-gather-daemonset-8sjs4" event={"ID":"3a94e4db-b581-4cf2-b961-4225304b0572","Type":"ContainerStarted","Data":"9c7950476fda9092cbd40b1fbac27d5e726958cafae4ae06746028fb615ebcdf"}
Oct 30 23:24:43.535923 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:24:43.535850    4477 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-qktfv/perf-node-gather-daemonset-8sjs4" podStartSLOduration=1.535838278 podStartE2EDuration="1.535838278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:24:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:24:43.533985565 +0000 UTC m=+4667.459911770" watchObservedRunningTime="2026-10-30 23:24:43.535838278 +0000 UTC m=+4667.461764528"
Oct 30 23:24:43.660719 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:24:43.660688    4477 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-bd84r_c0550643-5300-4f60-ad13-ac4504c60a13/kube-state-metrics/0.log"
Oct 30 23:24:43.773565 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:24:43.773533    4477 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-bd84r_c0550643-5300-4f60-ad13-ac4504c60a13/kube-rbac-proxy-main/0.log"
Oct 30 23:24:43.892461 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:24:43.892429    4477 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-bd84r_c0550643-5300-4f60-ad13-ac4504c60a13/kube-rbac-proxy-self/0.log"
Oct 30 23:24:44.013724 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:24:44.013698    4477 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-c78d588bd-vf9gh_c7a30fb6-9f07-4ade-82d3-d3f00f02476f/metrics-server/0.log"
Oct 30 23:24:44.132951 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:24:44.132928    4477 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-shq24_4dc9807e-ebe7-4789-b0db-6a09ce597ed2/monitoring-plugin/0.log"
Oct 30 23:24:44.615321 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:24:44.615296    4477 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7cwrt_46d31fbe-a529-46d2-9385-1bf960a9fb85/node-exporter/0.log"
Oct 30 23:24:44.734225 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:24:44.734188    4477 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7cwrt_46d31fbe-a529-46d2-9385-1bf960a9fb85/kube-rbac-proxy/0.log"
Oct 30 23:24:44.851651 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:24:44.851627    4477 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7cwrt_46d31fbe-a529-46d2-9385-1bf960a9fb85/init-textfile/0.log"
Oct 30 23:24:45.358138 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:24:45.358109    4477 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-p6d59_de5a375a-a179-450f-b21d-982bfa72e489/kube-rbac-proxy-main/0.log"
Oct 30 23:24:45.475440 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:24:45.475418    4477 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-p6d59_de5a375a-a179-450f-b21d-982bfa72e489/kube-rbac-proxy-self/0.log"
Oct 30 23:24:45.593954 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:24:45.593931    4477 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-p6d59_de5a375a-a179-450f-b21d-982bfa72e489/openshift-state-metrics/0.log"
Oct 30 23:24:45.723793 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:24:45.723769    4477 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_030c1cfb-02a0-4628-9451-970fae0a8113/prometheus/0.log"
Oct 30 23:24:45.841860 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:24:45.841836    4477 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_030c1cfb-02a0-4628-9451-970fae0a8113/config-reloader/0.log"
Oct 30 23:24:45.962952 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:24:45.962934    4477 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_030c1cfb-02a0-4628-9451-970fae0a8113/thanos-sidecar/0.log"
Oct 30 23:24:46.080606 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:24:46.080530    4477 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_030c1cfb-02a0-4628-9451-970fae0a8113/kube-rbac-proxy-web/0.log"
Oct 30 23:24:46.198744 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:24:46.198713    4477 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_030c1cfb-02a0-4628-9451-970fae0a8113/kube-rbac-proxy/0.log"
Oct 30 23:24:46.317978 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:24:46.317950    4477 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_030c1cfb-02a0-4628-9451-970fae0a8113/kube-rbac-proxy-thanos/0.log"
Oct 30 23:24:46.434726 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:24:46.434705    4477 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_030c1cfb-02a0-4628-9451-970fae0a8113/init-config-reloader/0.log"
Oct 30 23:24:46.559785 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:24:46.559764    4477 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-rcb97_4dc828b5-9f84-47db-95c9-24e3ae008789/prometheus-operator/0.log"
Oct 30 23:24:46.671954 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:24:46.671934    4477 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-rcb97_4dc828b5-9f84-47db-95c9-24e3ae008789/kube-rbac-proxy/0.log"
Oct 30 23:24:46.791230 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:24:46.791162    4477 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-q284j_819fb22e-2858-4cbe-9735-5f98f2b53cbc/prometheus-operator-admission-webhook/0.log"
Oct 30 23:24:46.914230 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:24:46.914204    4477 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-69dd59c556-ssvjv_beb3893f-2dc3-4c93-b03c-8d7c35e47a89/telemeter-client/0.log"
Oct 30 23:24:47.030601 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:24:47.030577    4477 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-69dd59c556-ssvjv_beb3893f-2dc3-4c93-b03c-8d7c35e47a89/reload/0.log"
Oct 30 23:24:47.148870 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:24:47.148843    4477 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-69dd59c556-ssvjv_beb3893f-2dc3-4c93-b03c-8d7c35e47a89/kube-rbac-proxy/0.log"
Oct 30 23:24:49.512673 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:24:49.512646    4477 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-qktfv/perf-node-gather-daemonset-8sjs4"
Oct 30 23:24:55.291226 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:24:55.291184    4477 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-c749j_9ea63ef4-6b84-46e6-a753-bca60280734a/dns/0.log"
Oct 30 23:24:55.410333 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:24:55.410303    4477 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-c749j_9ea63ef4-6b84-46e6-a753-bca60280734a/kube-rbac-proxy/0.log"
Oct 30 23:24:56.026323 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:24:56.026291    4477 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4knxq_da9b75bf-8af2-450f-9a98-c265eea48282/dns-node-resolver/0.log"
Oct 30 23:24:56.274659 ip-10-0-130-77 kubenswrapper[4477]: E1030 23:24:56.274626    4477 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-69blh" containerName="extractor"
Oct 30 23:24:57.562744 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:24:57.562714    4477 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-wx4cl_6f02c360-27f4-4a25-8155-436e260be559/discovery/0.log"
Oct 30 23:24:57.650224 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:24:57.650192    4477 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6f865f994d-vg2nd_5fc4bd88-0db5-4608-ab12-88364e90d0c9/kube-auth-proxy/0.log"
Oct 30 23:24:57.832637 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:24:57.832532    4477 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-bjfxc_eeb7af55-f99f-411e-9f71-8ac1638c67ed/istio-proxy/0.log"
Oct 30 23:24:57.869893 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:24:57.869863    4477 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-b74lk_e9c20fdf-bea5-4d6c-b252-b29b08bfdf63/node-ca/0.log"
Oct 30 23:24:57.889128 ip-10-0-130-77 kubenswrapper[4477]: I1030 23:24:57.889100    4477 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6db586dd6c-d8sgs_4d8945b9-b167-4397-bb13-a9ef77e413f7/istio-proxy/0.log"